public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v28] Index skip scan
58+ messages / 5 participants
[nested] [flat]
* [PATCH v28] Index skip scan
@ 2019-07-03 14:25 erthalion <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: erthalion @ 2019-07-03 14:25 UTC (permalink / raw)
Implementation of Index Skip Scan (see Loose Index Scan in the wiki [1])
on top of IndexOnlyScan and IndexScan. To make it suitable for both
situations when there are small number of distinct values and
significant amount of distinct values the following approach is taken -
instead of searching from the root for every value we're searching for
then first on the current page, and then if not found continue searching
from the root.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 +
doc/src/sgml/indexam.sgml | 63 +++
doc/src/sgml/indices.sgml | 24 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 +
src/backend/access/nbtree/nbtree.c | 13 +
src/backend/access/nbtree/nbtsearch.c | 359 ++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 29 +
src/backend/executor/nodeIndexonlyscan.c | 51 +-
src/backend/executor/nodeIndexscan.c | 51 +-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 3 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/pathkeys.c | 84 ++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/planagg.c | 1 +
src/backend/optimizer/plan/planner.c | 91 +++-
src/backend/optimizer/util/pathnode.c | 40 ++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/nbtree.h | 7 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 10 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 5 +
src/include/optimizer/paths.h | 4 +
src/test/regress/expected/create_index.out | 1 +
src/test/regress/expected/select_distinct.out | 511 ++++++++++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/create_index.sql | 2 +
src/test/regress/sql/select_distinct.sql | 196 +++++++
42 files changed, 1626 insertions(+), 25 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index cc1670934f..ab9f0a7177 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -129,6 +129,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 89284dc5c0..3edd12dd27 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4413,6 +4413,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index dd54c68802..73b1b4fcf7 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -144,6 +144,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -687,6 +688,68 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ ScanDirection indexdir,
+ bool scanstart,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>indexdir</parameter></term>
+ <listitem>
+ <para>
+ The index direction, in which data must be read.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>scanstart</parameter></term>
+ <listitem>
+ <para>
+ Whether or not it is a start of the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index 95c0a1926c..567141046f 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1235,6 +1235,30 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ In case if index scan is used to retrieve the distinct values of a column
+ efficiently, it can be not very efficient, since it requires to scan all
+ the equal values of a key. In such cases planner will consider apply index
+ skip scan approach, which is based on the idea of
+ <firstterm>Loose index scan</firstterm>. Rather than scanning all equal
+ values of a key, as soon as a new value is found, it will search for a
+ larger value on the same index page, and if not found, restart the search
+ by looking for a larger value. This is much faster when the index has many
+ equal keys.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index ae7b729edd..233ea9e5ec 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -109,6 +109,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index cf9699ad18..9817f34c34 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -61,6 +61,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0cc87911d6..38072ad24b 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -83,6 +83,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 5cc30dac42..019e330cff 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 28edd4aca7..ae7a882571 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -730,6 +731,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 4cfd5289ad..46471598d1 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -131,6 +131,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
+ amroutine->amskip = btskip;
amroutine->amcostestimate = btcostestimate;
amroutine->amoptions = btoptions;
amroutine->amproperty = btproperty;
@@ -380,6 +381,8 @@ btbeginscan(Relation rel, int nkeys, int norderbys)
*/
so->currTuples = so->markTuples = NULL;
+ so->skipScanKey = NULL;
+
scan->xs_itupdesc = RelationGetDescr(rel);
scan->opaque = so;
@@ -447,6 +450,16 @@ btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
_bt_preprocess_array_keys(scan);
}
+/*
+ * btskip() -- skip to the beginning of the next key prefix
+ */
+bool
+btskip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix)
+{
+ return _bt_skip(scan, direction, indexdir, start, prefix);
+}
+
/*
* btendscan() -- close down a scan
*/
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index 7f77ed24c5..d57de9dfa1 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -37,6 +37,10 @@ static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
static Buffer _bt_walk_left(Relation rel, Buffer buf, Snapshot snapshot);
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
+static inline void _bt_update_skip_scankeys(IndexScanDesc scan,
+ Relation indexRel);
+static inline bool _bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir);
/*
@@ -1373,6 +1377,305 @@ _bt_next(IndexScanDesc scan, ScanDirection dir)
return true;
}
+/*
+ * _bt_skip() -- Skip items that have the same prefix as the most recently
+ * fetched index tuple.
+ *
+ * The current position is set so that a subsequent call to _bt_next will
+ * fetch the first tuple that differs in the leading 'prefix' keys.
+ *
+ * There are four different kinds of skipping (depending on dir and
+ * indexdir, that are important to distinguish, especially in the presense
+ * of an index condition:
+ *
+ * * Advancing forward and reading forward
+ * simple scan
+ *
+ * * Advancing forward and reading backward
+ * scan inside a cursor fetching backward, when skipping is necessary
+ * right from the start
+ *
+ * * Advancing backward and reading forward
+ * scan with order by desc inside a cursor fetching forward, when
+ * skipping is necessary right from the start
+ *
+ * * Advancing backward and reading backward
+ * simple scan with order by desc
+ */
+bool
+_bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ BTStack stack;
+ Buffer buf;
+ OffsetNumber offnum;
+ BTScanPosItem *currItem;
+ Relation indexRel = scan->indexRelation;
+
+ /* We want to return tuples, and we need a starting point */
+ Assert(scan->xs_want_itup);
+ Assert(scan->xs_itup);
+
+ /*
+ * If skipScanKey is NULL then we initialize it with _bt_mkscankey,
+ * otherwise we will just update the sk_flags / sk_argument elements in
+ * order to eliminate repeated free/realloc.
+ */
+ if (so->skipScanKey == NULL)
+ {
+ so->skipScanKey = _bt_mkscankey(indexRel, scan->xs_itup);
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->scantid = NULL;
+ }
+ else
+ {
+ _bt_update_skip_scankeys(scan, indexRel);
+ }
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /* Check if the next unique key can be found within the current page */
+ if (BTScanPosIsValid(so->currPos) &&
+ _bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ {
+ bool keyFound = false;
+
+ LockBuffer(so->currPos.buf, BT_READ);
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, so->currPos.buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(so->currPos.buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /* Now read the data */
+ keyFound = _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (keyFound)
+ {
+ /* set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+ }
+ }
+
+ if (BTScanPosIsValid(so->currPos))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ /*
+ * We haven't found scan key within the current page, so let's scan from
+ * the root. Use _bt_search and _bt_binsrch to get the buffer and offset
+ * number
+ */
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /*
+ * Simplest case is when both directions are forward, when we are already
+ * at the next distinct key at the beginning of the series (so everything
+ * else would be done in _bt_readpage)
+ *
+ * The case when both directions are backwards is also simple, but we need
+ * to go one step back, since we need a last element from the previous
+ * series.
+ */
+ if (ScanDirectionIsBackward(dir) && ScanDirectionIsBackward(indexdir))
+ offnum = OffsetNumberPrev(offnum);
+
+ /*
+ * Andvance backward but read forward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can read forward without doing anything else. Otherwise
+ * find previous distinct key and the beginning of it's series and read
+ * forward from there. To do so, go back one step, perform binary search
+ * to find the first item in the series and let _bt_readpage do everything
+ * else.
+ */
+ else if (ScanDirectionIsBackward(dir) && ScanDirectionIsForward(indexdir))
+ {
+ if (!scanstart)
+ {
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* One step back to find a previous value */
+ _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (_bt_next(scan, dir))
+ {
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /*
+ * And now find the last item from the sequence for the
+ * current, value with the intention do OffsetNumberNext. As a
+ * result we end up on a first element from the sequence.
+ */
+ if (_bt_scankey_within_page(scan, so->skipScanKey,
+ so->currPos.buf, dir))
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ else
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ }
+ }
+ else
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ }
+
+ /*
+ * Andvance forward but read backward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can go one step back and read forward without doing
+ * anything else. Otherwise find the next distinct key and the beginning
+ * of it's series, go one step back and read backward from there.
+ *
+ * An interesting situation can happen if one of distinct keys do not pass
+ * a corresponding index condition at all. In this case reading backward
+ * can lead to a previous distinc key being found, creating a loop. To
+ * avoid that check the value to be returned, and jump one more time if
+ * it's the same as at the beginning.
+ */
+ else if (ScanDirectionIsForward(dir) && ScanDirectionIsBackward(indexdir))
+ {
+ if (scanstart)
+ offnum = OffsetNumberPrev(offnum);
+ else
+ {
+ OffsetNumber nextOffset,
+ startOffset;
+
+ nextOffset = startOffset = ItemPointerGetOffsetNumber(&scan->xs_itup->t_tid);
+
+ while (nextOffset == startOffset)
+ {
+ /*
+ * Find a next index tuple to update scan key. It could be at
+ * the end, so check for max offset
+ */
+ OffsetNumber curOffnum = offnum;
+ Page page = BufferGetPage(so->currPos.buf);
+ OffsetNumber maxoff = PageGetMaxOffsetNumber(page);
+ ItemId itemid = PageGetItemId(page, Min(offnum, maxoff));
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ scan->xs_itup = (IndexTuple) PageGetItem(page, itemid);
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ if (BTScanPosIsValid(so->currPos))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /*
+ * Jump to the next key returned the same offset, which means
+ * we are at the end and need to return
+ */
+ if (offnum == curOffnum)
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+
+ offnum = OffsetNumberPrev(offnum);
+
+ /* Check if _bt_readpage returns already found item */
+ if (_bt_readpage(scan, indexdir, offnum))
+ {
+ IndexTuple itup;
+
+ currItem = &so->currPos.items[so->currPos.lastItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ nextOffset = ItemPointerGetOffsetNumber(&itup->t_tid);
+ }
+ else
+ elog(ERROR, "Could not read closest index tuples: %d", offnum);
+
+ /*
+ * If the nextOffset is the same as before, it means we are in
+ * the loop, return offnum to the original position and jump
+ * further
+ */
+ if (nextOffset == startOffset)
+ offnum = OffsetNumberNext(offnum);
+ }
+ }
+ }
+
+ /* Now read the data */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to advance to
+ * the next page. Return false if there's no matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ else
+ {
+ /* Drop the lock, and maybe the pin, on the current page */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ /* And set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ return true;
+}
+
/*
* _bt_readpage() -- Load data from current index page into so->currPos
*
@@ -2244,3 +2547,59 @@ _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
so->numKilled = 0; /* just paranoia */
so->markItemIndex = -1; /* ditto */
}
+
+/*
+ * _bt_update_skip_scankeys() -- set up a new values for the existing scankeys
+ * based on the current index tuple
+ */
+static inline void
+_bt_update_skip_scankeys(IndexScanDesc scan, Relation indexRel)
+{
+ TupleDesc itupdesc;
+ int indnkeyatts,
+ i;
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ ScanKey scankeys = so->skipScanKey->scankeys;
+
+ itupdesc = RelationGetDescr(indexRel);
+ indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ for (i = 0; i < indnkeyatts; i++)
+ {
+ Datum datum;
+ bool null;
+ int flags;
+
+ datum = index_getattr(scan->xs_itup, i + 1, itupdesc, &null);
+ flags = (null ? SK_ISNULL : 0) |
+ (indexRel->rd_indoption[i] << SK_BT_INDOPTION_SHIFT);
+ scankeys[i].sk_flags = flags;
+ scankeys[i].sk_argument = datum;
+ }
+}
+
+/*
+ * _bt_scankey_within_page() -- check if the provided scankey could be found
+ * within a page, specified by the buffer.
+ */
+static inline bool
+_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir)
+{
+ OffsetNumber low,
+ high,
+ compare_offset;
+ Page page = BufferGetPage(buf);
+ BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ int compare_value = ScanDirectionIsForward(dir) ? 0 : 1;
+
+ low = P_FIRSTDATAKEY(opaque);
+ high = PageGetMaxOffsetNumber(page);
+
+ if (unlikely(high < low))
+ return false;
+
+ compare_offset = ScanDirectionIsForward(dir) ? high : low;
+
+ return _bt_compare(scan->indexRelation,
+ key, page, compare_offset) > compare_value;
+}
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 45472db147..dc151ecf09 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -64,6 +64,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 62fb3434a3..ad500de12b 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -130,6 +130,7 @@ static void ExplainDummyGroup(const char *objtype, const char *labelname,
static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(ExplainState *es, int skipPrefixSize);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1041,6 +1042,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(ExplainState *es, int skipPrefixSize)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1363,6 +1380,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(es, indexscan->indexskipprefixsize);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1373,6 +1392,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(es, indexonlyscan->indexskipprefixsize);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1582,6 +1603,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ {
+ ExplainPropertyBool("Skip scan mode", true, es);
+ }
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1595,6 +1620,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ {
+ ExplainPropertyBool("Skip scan mode", true, es);
+ }
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 652a9afc75..2aae3daae4 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -65,6 +65,13 @@ IndexOnlyNext(IndexOnlyScanState *node)
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +79,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,14 +122,50 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, direction)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* We can skip the heap fetch if the TID references a heap page on
@@ -250,6 +293,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -500,6 +545,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index ac7aa81f67..3a7f5e6b8b 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -116,6 +123,7 @@ IndexNext(IndexScanState *node)
node->iss_NumOrderByKeys);
node->iss_ScanDesc = scandesc;
+ node->iss_ScanDesc->xs_want_itup = true;
/*
* If no run-time keys to calculate or they are ready, go ahead and
@@ -127,12 +135,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +193,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -906,6 +951,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index a2617c7cfd..20495c9e52 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -490,6 +490,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -515,6 +516,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index e6ce8e2110..2ff9625533 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -559,6 +559,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -573,6 +574,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -2213,6 +2215,7 @@ _outPlannerInfo(StringInfo str, const PlannerInfo *node)
WRITE_NODE_FIELD(group_pathkeys);
WRITE_NODE_FIELD(window_pathkeys);
WRITE_NODE_FIELD(distinct_pathkeys);
+ WRITE_NODE_FIELD(uniq_distinct_pathkeys);
WRITE_NODE_FIELD(sort_pathkeys);
WRITE_NODE_FIELD(processed_tlist);
WRITE_NODE_FIELD(minmax_aggs);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 764e3bb90c..0fc3c5ea68 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1787,6 +1787,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1806,6 +1807,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index c5f6593485..194e258dc1 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -124,6 +124,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 2f4fea241a..70c1df47a4 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -29,6 +29,7 @@
#include "utils/lsyscache.h"
+static bool pathkey_is_unique(PathKey *new_pathkey, List *pathkeys);
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
RelOptInfo *partrel,
@@ -96,6 +97,30 @@ make_canonical_pathkey(PlannerInfo *root,
return pk;
}
+/*
+ * pathkey_is_unique
+ * Part of pathkey_is_redundant, that is reponsible for the case, when the
+ * new pathkey's equivalence class is the same as that of any existing
+ * member of the pathkey list.
+ */
+static bool
+pathkey_is_unique(PathKey *new_pathkey, List *pathkeys)
+{
+ EquivalenceClass *new_ec = new_pathkey->pk_eclass;
+ ListCell *lc;
+
+ /* If same EC already used in list, then redundant */
+ foreach(lc, pathkeys)
+ {
+ PathKey *old_pathkey = (PathKey *) lfirst(lc);
+
+ if (new_ec == old_pathkey->pk_eclass)
+ return true;
+ }
+
+ return false;
+}
+
/*
* pathkey_is_redundant
* Is a pathkey redundant with one already in the given list?
@@ -135,22 +160,12 @@ static bool
pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys)
{
EquivalenceClass *new_ec = new_pathkey->pk_eclass;
- ListCell *lc;
/* Check for EC containing a constant --- unconditionally redundant */
if (EC_MUST_BE_REDUNDANT(new_ec))
return true;
- /* If same EC already used in list, then redundant */
- foreach(lc, pathkeys)
- {
- PathKey *old_pathkey = (PathKey *) lfirst(lc);
-
- if (new_ec == old_pathkey->pk_eclass)
- return true;
- }
-
- return false;
+ return pathkey_is_unique(new_pathkey, pathkeys);
}
/*
@@ -1098,6 +1113,53 @@ make_pathkeys_for_sortclauses(PlannerInfo *root,
return pathkeys;
}
+/*
+ * make_pathkeys_for_distinctclauses
+ * Generate a pathkeys list for distinct clauses, that represents the sort
+ * order specified by a list of SortGroupClauses. Similar to
+ * make_pathkeys_for_sortclauses, but allows to specify if we need to
+ * check the full redundancy, or just uniqueness.
+ */
+List *
+make_pathkeys_for_distinctclauses(PlannerInfo *root,
+ List *distinctclauses,
+ List *tlist, bool checkRedundant)
+{
+ List *pathkeys = NIL;
+ ListCell *l;
+
+ foreach(l, distinctclauses)
+ {
+ SortGroupClause *sortcl = (SortGroupClause *) lfirst(l);
+ Expr *sortkey;
+ PathKey *pathkey;
+
+ sortkey = (Expr *) get_sortgroupclause_expr(sortcl, tlist);
+ Assert(OidIsValid(sortcl->sortop));
+ pathkey = make_pathkey_from_sortop(root,
+ sortkey,
+ root->nullable_baserels,
+ sortcl->sortop,
+ sortcl->nulls_first,
+ sortcl->tleSortGroupRef,
+ true);
+
+ /* Canonical form eliminates redundant ordering keys */
+ if (checkRedundant)
+ {
+ if (!pathkey_is_redundant(pathkey, pathkeys))
+ pathkeys = lappend(pathkeys, pathkey);
+ }
+ else
+ {
+ if (!pathkey_is_unique(pathkey, pathkeys))
+ pathkeys = lappend(pathkeys, pathkey);
+ }
+ }
+ return pathkeys;
+}
+
+
/****************************************************************************
* PATHKEYS AND MERGECLAUSES
****************************************************************************/
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 0c036209f0..6e54446b29 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -175,12 +175,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2905,7 +2907,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2916,7 +2919,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5179,7 +5183,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5196,6 +5201,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5208,7 +5214,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5223,6 +5230,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/plan/planagg.c b/src/backend/optimizer/plan/planagg.c
index 9381939c82..ed52139839 100644
--- a/src/backend/optimizer/plan/planagg.c
+++ b/src/backend/optimizer/plan/planagg.c
@@ -505,6 +505,7 @@ minmax_qp_callback(PlannerInfo *root, void *extra)
root->group_pathkeys = NIL;
root->window_pathkeys = NIL;
root->distinct_pathkeys = NIL;
+ root->uniq_distinct_pathkeys = NIL;
root->sort_pathkeys =
make_pathkeys_for_sortclauses(root,
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 17c5f086fb..644b8ad356 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -3622,12 +3622,21 @@ standard_qp_callback(PlannerInfo *root, void *extra)
if (parse->distinctClause &&
grouping_is_sortable(parse->distinctClause))
+ {
+ root->uniq_distinct_pathkeys =
+ make_pathkeys_for_distinctclauses(root,
+ parse->distinctClause,
+ tlist, false);
root->distinct_pathkeys =
- make_pathkeys_for_sortclauses(root,
- parse->distinctClause,
- tlist);
+ make_pathkeys_for_distinctclauses(root,
+ parse->distinctClause,
+ tlist, true);
+ }
else
+ {
root->distinct_pathkeys = NIL;
+ root->uniq_distinct_pathkeys = NIL;
+ }
root->sort_pathkeys =
make_pathkeys_for_sortclauses(root,
@@ -4814,6 +4823,82 @@ create_distinct_paths(PlannerInfo *root,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+
+ /* Consider index skip scan as well */
+ if (enable_indexskipscan &&
+ IsA(path, IndexPath) &&
+ ((IndexPath *) path)->indexinfo->amcanskip &&
+ root->distinct_pathkeys != NIL)
+ {
+ ListCell *lc;
+ IndexOptInfo *index = NULL;
+ bool different_columns_order = false,
+ not_empty_qual = false;
+ int i = 0;
+ int distinctPrefixKeys;
+
+ Assert(path->pathtype == T_IndexOnlyScan ||
+ path->pathtype == T_IndexScan);
+
+ index = ((IndexPath *) path)->indexinfo;
+ distinctPrefixKeys = list_length(root->uniq_distinct_pathkeys);
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, root->uniq_distinct_pathkeys)
+ {
+ PathKey *pathKey = lfirst_node(PathKey, lc);
+ EquivalenceMember *em =
+ lfirst_node(EquivalenceMember,
+ list_head(pathKey->pk_eclass->ec_members));
+ Var *var = (Var *) em->em_expr;
+
+ Assert(i < index->ncolumns);
+
+ for (i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ /*
+ * XXX: In case of index scan quals evaluation happens
+ * after ExecScanFetch, which means skip results could be
+ * fitered out. Consider the following query:
+ *
+ * select distinct (a, b) a, b, c from t where c < 100;
+ *
+ * Skip scan returns one tuple for one distinct set of (a,
+ * b) with arbitrary one of c, so if the choosed c does
+ * not match the qual and there is any c that matches the
+ * qual, we miss that tuple.
+ */
+ if (path->pathtype == T_IndexScan &&
+ parse->jointree != NULL &&
+ parse->jointree->quals != NULL &&
+ list_length((List *) parse->jointree->quals) != 0)
+ not_empty_qual = true;
+
+ if (!different_columns_order && !not_empty_qual)
+ {
+ add_path(distinct_rel, (Path *)
+ create_skipscan_unique_path(root,
+ distinct_rel,
+ path,
+ distinctPrefixKeys,
+ numDistinctRows));
+ }
+ }
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 34acb732ee..e248584fb1 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2904,6 +2904,46 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *basepath,
+ int distinctPrefixKeys,
+ double numGroups)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /* The size of the prefix we'll use for skipping. */
+ Assert(pathnode->indexinfo->amcanskip);
+ Assert(distinctPrefixKeys > 0);
+ /*Assert(distinctPrefixKeys <= list_length(pathnode->path.pathkeys));*/
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ /*
+ * The cost to skip to each distinct value should be roughly the same as
+ * the cost of finding the first key times the number of distinct values
+ * we expect to find.
+ */
+ pathnode->path.startup_cost = basepath->startup_cost;
+ pathnode->path.total_cost = basepath->startup_cost * numGroups;
+ pathnode->path.rows = numGroups;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index cf1761401d..34fbc27716 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 90ffd89339..9e5b74b6de 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -910,6 +910,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3a61..88f9890780 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -351,6 +351,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 6e3db06eed..fd1595d3be 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -225,6 +232,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 8c053be2ca..e5ec5b07c8 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -173,6 +173,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 52eafe6b00..c1d6ff98f5 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -662,6 +662,9 @@ typedef struct BTScanOpaqueData
*/
int markItemIndex; /* itemIndex, or -1 if not valid */
+ /* Work space for _bt_skip */
+ BTScanInsert skipScanKey; /* used to control skipping */
+
/* keep these last in struct for efficiency */
BTScanPosData currPos; /* current position data */
BTScanPosData markPos; /* marked position, if any */
@@ -776,6 +779,8 @@ extern OffsetNumber _bt_binsrch_insert(Relation rel, BTInsertState insertstate);
extern int32 _bt_compare(Relation rel, BTScanInsert key, Page page, OffsetNumber offnum);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
+extern bool _bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
Snapshot snapshot);
@@ -800,6 +805,8 @@ extern void _bt_end_vacuum_callback(int code, Datum arg);
extern Size BTreeShmemSize(void);
extern void BTreeShmemInit(void);
extern bytea *btoptions(Datum reloptions, bool validate);
+extern bool btskip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 664e72ef5d..dff90fada1 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index f42189d2bf..93f9cdc33d 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1377,6 +1377,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1406,6 +1408,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1424,6 +1428,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 23a06d718e..3a21118ad1 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -298,6 +298,11 @@ struct PlannerInfo
List *group_pathkeys; /* groupClause pathkeys, if any */
List *window_pathkeys; /* pathkeys of bottom window, if any */
List *distinct_pathkeys; /* distinctClause pathkeys, if any */
+ List *uniq_distinct_pathkeys; /* unique, but potentially redundant
+ * distinctClause pathkeys, if any.
+ * Used for index skip scan, since
+ * redundant distinctClauses also must
+ * be considered */
List *sort_pathkeys; /* sortClause pathkeys, if any */
List *part_schemes; /* Canonicalised partition schemes used in the
@@ -833,6 +838,7 @@ struct IndexOptInfo
bool amsearchnulls; /* can AM search for NULL/NOT NULL entries? */
bool amhasgettuple; /* does AM have amgettuple interface? */
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
+ bool amcanskip; /* can AM skip duplicate values? */
bool amcanparallel; /* does AM support parallel scan? */
/* Rather than include amapi.h here, we declare amcostestimate like this */
void (*amcostestimate) (); /* AM's cost estimator */
@@ -1169,6 +1175,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1181,6 +1190,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 8e6594e355..f09c8c43a3 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -405,6 +405,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -432,6 +434,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index b3d0b4f6fb..9abfdfb6bd 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index a12af54971..7edcf4e689 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -200,6 +200,11 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *subpath,
+ int numCols,
+ double numGroups);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 7345137d1d..a782d12a50 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -209,6 +209,10 @@ extern List *build_join_pathkeys(PlannerInfo *root,
extern List *make_pathkeys_for_sortclauses(PlannerInfo *root,
List *sortclauses,
List *tlist);
+extern List *make_pathkeys_for_distinctclauses(PlannerInfo *root,
+ List *sortclauses,
+ List *tlist,
+ bool checkRedundant);
extern void initialize_mergeclause_eclasses(PlannerInfo *root,
RestrictInfo *restrictinfo);
extern void update_mergeclause_eclasses(PlannerInfo *root,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index c6d575a2f9..4f5c82f49d 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -19,6 +19,7 @@ CREATE INDEX tenk1_unique1 ON tenk1 USING btree(unique1 int4_ops);
CREATE INDEX tenk1_unique2 ON tenk1 USING btree(unique2 int4_ops);
CREATE INDEX tenk1_hundred ON tenk1 USING btree(hundred int4_ops);
CREATE INDEX tenk1_thous_tenthous ON tenk1 (thousand, tenthous);
+CREATE INDEX tenk1_four ON tenk1 (four);
CREATE INDEX tenk2_unique1 ON tenk2 USING btree(unique1 int4_ops);
CREATE INDEX tenk2_unique2 ON tenk2 USING btree(unique2 int4_ops);
CREATE INDEX tenk2_hundred ON tenk2 USING btree(hundred int4_ops);
diff --git a/src/test/regress/expected/select_distinct.out b/src/test/regress/expected/select_distinct.out
index f3696c6d1d..f0e92a99dd 100644
--- a/src/test/regress/expected/select_distinct.out
+++ b/src/test/regress/expected/select_distinct.out
@@ -244,3 +244,514 @@ SELECT null IS NOT DISTINCT FROM null as "yes";
t
(1 row)
+-- index only skip scan
+SELECT DISTINCT four FROM tenk1;
+ four
+------
+ 0
+ 1
+ 2
+ 3
+(4 rows)
+
+SELECT DISTINCT four FROM tenk1 WHERE four = 1;
+ four
+------
+ 1
+(1 row)
+
+SELECT DISTINCT four FROM tenk1 ORDER BY four DESC;
+ four
+------
+ 3
+ 2
+ 1
+ 0
+(4 rows)
+
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, hundred, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) hundred
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+FETCH FROM c;
+ a | b
+---+---
+ 1 | 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+END;
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+FETCH FROM c;
+ a | b
+---+-------
+ 5 | 10000
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+END;
+DROP TABLE distinct_a;
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+ QUERY PLAN
+--------------------------------------------------------------
+ Index Only Scan using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan mode: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 1 | 2
+ 3 | 1 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 1 | 2
+ 1 | 1 | 2
+(2 rows)
+
+END;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+ QUERY PLAN
+-----------------------------------------------------------------------
+ Index Only Scan Backward using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan mode: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 2 | 2
+ 1 | 2 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 2 | 2
+ 3 | 2 | 2
+(2 rows)
+
+END;
+DROP TABLE distinct_abc;
+-- index skip scan
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 ORDER BY four;
+ four | ten
+------+-----
+ 0 | 0
+ 1 | 9
+ 2 | 0
+ 3 | 1
+(4 rows)
+
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 WHERE four = 1 ORDER BY four;
+ four | ten
+------+-----
+ 1 | 9
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 ORDER BY four;
+ QUERY PLAN
+--------------------------------------
+ Index Scan using tenk1_four on tenk1
+ Skip scan mode: true
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 WHERE four = 1 ORDER BY four;
+ QUERY PLAN
+---------------------------------------------------
+ Result
+ -> Unique
+ -> Bitmap Heap Scan on tenk1
+ Recheck Cond: (four = 1)
+ -> Bitmap Index Scan on tenk1_four
+ Index Cond: (four = 1)
+(6 rows)
+
+-- check colums order
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+ four
+------
+ 0
+ 2
+(2 rows)
+
+CREATE INDEX tenk1_four_ten on tenk1 (four, ten);
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+ four
+------
+ 0
+ 2
+(2 rows)
+
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE four = 0;
+ four | ten
+------+-----
+ 0 | 0
+ 0 | 2
+ 0 | 4
+ 0 | 6
+ 0 | 8
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+ QUERY PLAN
+-----------------------------------------------
+ Index Only Scan using tenk1_four_ten on tenk1
+ Skip scan mode: true
+ Index Cond: (ten = 2)
+(3 rows)
+
+-- test uniq_distinct_pathkeys
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE four = 0;
+ QUERY PLAN
+-----------------------------------------------
+ Index Only Scan using tenk1_four_ten on tenk1
+ Skip scan mode: true
+ Index Cond: (four = 0)
+(3 rows)
+
+DROP INDEX tenk1_four_ten;
+CREATE INDEX tenk1_ten_four on tenk1 (ten, four);
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+ four
+------
+ 0
+ 2
+(2 rows)
+
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE ten = 2;
+ four | ten
+------+-----
+ 0 | 2
+ 2 | 2
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+ QUERY PLAN
+-----------------------------------------------
+ Index Only Scan using tenk1_ten_four on tenk1
+ Skip scan mode: true
+ Index Cond: (ten = 2)
+(3 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE ten = 2;
+ QUERY PLAN
+-----------------------------------------------
+ Index Only Scan using tenk1_ten_four on tenk1
+ Skip scan mode: true
+ Index Cond: (ten = 2)
+(3 rows)
+
+DROP INDEX tenk1_ten_four;
+-- check projection case
+SELECT DISTINCT four, four FROM tenk1 WHERE ten = 2;
+ four | four
+------+------
+ 0 | 0
+ 2 | 2
+(2 rows)
+
+SELECT DISTINCT four, 1 FROM tenk1 WHERE ten = 2;
+ four | ?column?
+------+----------
+ 2 | 1
+ 0 | 1
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT four FROM tenk1;
+ QUERY PLAN
+-------------------------------------------
+ Index Only Scan using tenk1_four on tenk1
+ Skip scan mode: true
+(2 rows)
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT four FROM tenk1;
+FETCH FROM c;
+ four
+------
+ 0
+(1 row)
+
+FETCH BACKWARD FROM c;
+ four
+------
+(0 rows)
+
+FETCH 5 FROM c;
+ four
+------
+ 0
+ 1
+ 2
+ 3
+(4 rows)
+
+FETCH BACKWARD 5 FROM c;
+ four
+------
+ 3
+ 2
+ 1
+ 0
+(4 rows)
+
+FETCH 5 FROM c;
+ four
+------
+ 0
+ 1
+ 2
+ 3
+(4 rows)
+
+FETCH BACKWARD 5 FROM c;
+ four
+------
+ 3
+ 2
+ 1
+ 0
+(4 rows)
+
+END;
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 9999
+ 1 | 10000
+(5 rows)
+
+DROP TABLE distinct_visibility;
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Index Only Scan using distinct_boundaries_a_b_c_idx on distinct_boundaries
+ Skip scan mode: true
+ Index Cond: ((b >= 1) AND (c = 0))
+(3 rows)
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ a | b | c
+---+---+---
+ 1 | 2 | 0
+ 2 | 2 | 0
+ 3 | 2 | 0
+ 4 | 2 | 0
+ 5 | 2 | 0
+(5 rows)
+
+DROP TABLE distinct_boundaries;
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a1c90eb905..bd3b373515 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -78,6 +78,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_hashjoin | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -89,7 +90,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(17 rows)
+(18 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index f96bebf410..a3be42a725 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -26,6 +26,8 @@ CREATE INDEX tenk1_hundred ON tenk1 USING btree(hundred int4_ops);
CREATE INDEX tenk1_thous_tenthous ON tenk1 (thousand, tenthous);
+CREATE INDEX tenk1_four ON tenk1 (four);
+
CREATE INDEX tenk2_unique1 ON tenk2 USING btree(unique1 int4_ops);
CREATE INDEX tenk2_unique2 ON tenk2 USING btree(unique2 int4_ops);
diff --git a/src/test/regress/sql/select_distinct.sql b/src/test/regress/sql/select_distinct.sql
index a605e86449..fddd0256ff 100644
--- a/src/test/regress/sql/select_distinct.sql
+++ b/src/test/regress/sql/select_distinct.sql
@@ -73,3 +73,199 @@ SELECT 1 IS NOT DISTINCT FROM 2 as "no";
SELECT 2 IS NOT DISTINCT FROM 2 as "yes";
SELECT 2 IS NOT DISTINCT FROM null as "no";
SELECT null IS NOT DISTINCT FROM null as "yes";
+
+-- index only skip scan
+SELECT DISTINCT four FROM tenk1;
+SELECT DISTINCT four FROM tenk1 WHERE four = 1;
+SELECT DISTINCT four FROM tenk1 ORDER BY four DESC;
+
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, hundred, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) hundred
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+DROP TABLE distinct_a;
+
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+DROP TABLE distinct_abc;
+
+-- index skip scan
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 ORDER BY four;
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 WHERE four = 1 ORDER BY four;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 ORDER BY four;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (four) four, ten
+FROM tenk1 WHERE four = 1 ORDER BY four;
+
+-- check colums order
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+
+CREATE INDEX tenk1_four_ten on tenk1 (four, ten);
+
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE four = 0;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+
+-- test uniq_distinct_pathkeys
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE four = 0;
+
+DROP INDEX tenk1_four_ten;
+
+CREATE INDEX tenk1_ten_four on tenk1 (ten, four);
+
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE ten = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT four FROM tenk1 WHERE ten = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (four, ten) four, ten FROM tenk1 WHERE ten = 2;
+
+DROP INDEX tenk1_ten_four;
+
+-- check projection case
+SELECT DISTINCT four, four FROM tenk1 WHERE ten = 2;
+SELECT DISTINCT four, 1 FROM tenk1 WHERE ten = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT four FROM tenk1;
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT four FROM tenk1;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 5 FROM c;
+FETCH BACKWARD 5 FROM c;
+
+FETCH 5 FROM c;
+FETCH BACKWARD 5 FROM c;
+
+END;
+
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DROP TABLE distinct_visibility;
+
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+DROP TABLE distinct_boundaries;
--
2.21.0
--72mz6bxvro3reh33--
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v32 2/2] Index skip scan
@ 2019-11-15 14:46 jesperpedersen <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: jesperpedersen @ 2019-11-15 14:46 UTC (permalink / raw)
Implementation of Index Skip Scan (see Loose Index Scan in the wiki [1])
on top of IndexOnlyScan and IndexScan. To make it suitable for both
situations when there are small number of distinct values and
significant amount of distinct values the following approach is taken -
instead of searching from the root for every value we're searching for
then first on the current page, and then if not found continue searching
from the root.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 +
doc/src/sgml/indexam.sgml | 63 ++
doc/src/sgml/indices.sgml | 23 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 +
src/backend/access/nbtree/nbtree.c | 13 +
src/backend/access/nbtree/nbtsearch.c | 466 +++++++++++++-
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +
src/backend/executor/nodeIndexonlyscan.c | 97 ++-
src/backend/executor/nodeIndexscan.c | 56 +-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/planner.c | 76 +++
src/backend/optimizer/util/pathnode.c | 39 ++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/nbtree.h | 7 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 5 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 5 +
src/test/regress/expected/select_distinct.out | 601 ++++++++++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/select_distinct.sql | 248 ++++++++
37 files changed, 1820 insertions(+), 12 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 0104d02f67..a018b7f3d0 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -133,6 +133,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e07dc01e80..36ba75b077 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4517,6 +4517,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 37f8d8760a..a726d80878 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -148,6 +148,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -691,6 +692,68 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ ScanDirection indexdir,
+ bool scanstart,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>indexdir</parameter></term>
+ <listitem>
+ <para>
+ The index direction, in which data must be read.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>scanstart</parameter></term>
+ <listitem>
+ <para>
+ Whether or not it is a start of the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index c54bf0dbbd..c429d98fc7 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1254,6 +1254,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 2e8f67ef10..4db31bb211 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -113,6 +113,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a7e55caf28..8dd1d30d2a 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -65,6 +65,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index aefc302ed2..8c692f7fb4 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 4871b7ff4d..e5fa4c7864 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 01539b6bd6..1047a35ade 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -730,6 +731,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 5254bc7ef5..8fde56fe60 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -132,6 +132,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
+ amroutine->amskip = btskip;
amroutine->amcostestimate = btcostestimate;
amroutine->amoptions = btoptions;
amroutine->amproperty = btproperty;
@@ -381,6 +382,8 @@ btbeginscan(Relation rel, int nkeys, int norderbys)
*/
so->currTuples = so->markTuples = NULL;
+ so->skipScanKey = NULL;
+
scan->xs_itupdesc = RelationGetDescr(rel);
scan->opaque = so;
@@ -448,6 +451,16 @@ btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
_bt_preprocess_array_keys(scan);
}
+/*
+ * btskip() -- skip to the beginning of the next key prefix
+ */
+bool
+btskip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix)
+{
+ return _bt_skip(scan, direction, indexdir, start, prefix);
+}
+
/*
* btendscan() -- close down a scan
*/
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index c573814f01..53518c6fdb 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -37,7 +37,10 @@ static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
static Buffer _bt_walk_left(Relation rel, Buffer buf, Snapshot snapshot);
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
-
+static inline void _bt_update_skip_scankeys(IndexScanDesc scan,
+ Relation indexRel);
+static inline bool _bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir);
/*
* _bt_drop_lock_and_maybe_pin()
@@ -1375,6 +1378,416 @@ _bt_next(IndexScanDesc scan, ScanDirection dir)
return true;
}
+/*
+ * _bt_skip() -- Skip items that have the same prefix as the most recently
+ * fetched index tuple.
+ *
+ * The current position is set so that a subsequent call to _bt_next will
+ * fetch the first tuple that differs in the leading 'prefix' keys.
+ *
+ * There are four different kinds of skipping (depending on dir and
+ * indexdir, that are important to distinguish, especially in the presense
+ * of an index condition:
+ *
+ * * Advancing forward and reading forward
+ * simple scan
+ *
+ * * Advancing forward and reading backward
+ * scan inside a cursor fetching backward, when skipping is necessary
+ * right from the start
+ *
+ * * Advancing backward and reading forward
+ * scan with order by desc inside a cursor fetching forward, when
+ * skipping is necessary right from the start
+ *
+ * * Advancing backward and reading backward
+ * simple scan with order by desc
+ *
+ * The current page is searched for the next unique value. If none is found
+ * we will do a scan from the root in order to find the next page with
+ * a unique value.
+ */
+bool
+_bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ BTStack stack;
+ Buffer buf;
+ OffsetNumber offnum;
+ BTScanPosItem *currItem;
+ Relation indexRel = scan->indexRelation;
+
+ /* We want to return tuples, and we need a starting point */
+ Assert(scan->xs_want_itup);
+ Assert(scan->xs_itup);
+
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ /* If skipScanKey is NULL then we initialize it with _bt_mkscankey */
+ if (so->skipScanKey == NULL)
+ {
+ so->skipScanKey = _bt_mkscankey(indexRel, scan->xs_itup);
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->scantid = NULL;
+ }
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /* Check if the next unique key can be found within the current page.
+ * Since we do not lock the current page between jumps, it's possible
+ * that it was splitted since the last time we saw it. This is fine in
+ * case of scanning forward, since page split to the right and we are
+ * still on the left most page. In case of scanning backwards it's
+ * possible to loose some pages and we need to remember the previous
+ * page, and then follow the right link from the current page until we
+ * find the original one.
+ *
+ * Since the whole idea of checking the current page is to protect
+ * ourselves and make more performant statistic mismatch case when
+ * there are too many distinct values for jumping, it's not clear if
+ * the complexity of this solution in case of backward scan is
+ * justified, so for now just avoid it.
+ */
+ if (BufferIsValid(so->currPos.buf) && ScanDirectionIsForward(dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ {
+ bool keyFound = false;
+
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, so->currPos.buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(so->currPos.buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /* Now read the data */
+ keyFound = _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+
+ if (keyFound)
+ {
+ /* set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+ }
+ }
+ else
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+ }
+
+ if (BufferIsValid(so->currPos.buf))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ /*
+ * We haven't found scan key within the current page, so let's scan from
+ * the root. Use _bt_search and _bt_binsrch to get the buffer and offset
+ * number
+ */
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /*
+ * Simplest case is when both directions are forward, when we are already
+ * at the next distinct key at the beginning of the series (so everything
+ * else would be done in _bt_readpage)
+ *
+ * The case when both directions are backwards is also simple, but we need
+ * to go one step back, since we need a last element from the previous
+ * series.
+ */
+ if (ScanDirectionIsBackward(dir) && ScanDirectionIsBackward(indexdir))
+ offnum = OffsetNumberPrev(offnum);
+
+ /*
+ * Andvance backward but read forward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can read forward without doing anything else. Otherwise
+ * find previous distinct key and the beginning of it's series and read
+ * forward from there. To do so, go back one step, perform binary search
+ * to find the first item in the series and let _bt_readpage do everything
+ * else.
+ */
+ else if (ScanDirectionIsBackward(dir) && ScanDirectionIsForward(indexdir))
+ {
+ if (!scanstart)
+ {
+ /* Reading forward means we expect to see more data on the right */
+ so->currPos.moreRight = true;
+
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* One step back to find a previous value */
+ _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (_bt_next(scan, dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /*
+ * And now find the last item from the sequence for the
+ * current, value with the intention do OffsetNumberNext. As a
+ * result we end up on a first element from the sequence.
+ */
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ else
+ {
+ if (BufferIsValid(so->currPos.buf))
+ {
+ /* Before leaving current page, deal with any killed items */
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ }
+ }
+ else
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ }
+
+ /*
+ * Advance forward but read backward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can go one step back and read forward without doing
+ * anything else. Otherwise find the next distinct key and the beginning
+ * of it's series, go one step back and read backward from there.
+ *
+ * An interesting situation can happen if one of distinct keys do not pass
+ * a corresponding index condition at all. In this case reading backward
+ * can lead to a previous distinct key being found, creating a loop. To
+ * avoid that check the value to be returned, and jump one more time if
+ * it's the same as at the beginning. Note that we do not check visibility
+ * here, and dead tuples could also lead to the same situation. This has to
+ * be checked on the caller side.
+ */
+ else if (ScanDirectionIsForward(dir) && ScanDirectionIsBackward(indexdir))
+ {
+ if (scanstart)
+ offnum = OffsetNumberPrev(offnum);
+ else
+ {
+ OffsetNumber nextOffset,
+ startOffset,
+ jumpOffset;
+
+ IndexTuple startItup = CopyIndexTuple(scan->xs_itup);
+ Page page = BufferGetPage(so->currPos.buf);
+
+ /* We are at the end and need to return */
+ if ((offnum > PageGetMaxOffsetNumber(page)) &
+ (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+
+ nextOffset = startOffset = ItemPointerGetOffsetNumber(&scan->xs_itup->t_tid);
+
+ /* Reading backwards means we expect to see more data on the left */
+ so->currPos.moreLeft = true;
+
+ while (nextOffset == startOffset)
+ {
+ IndexTuple itup;
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * Find a next index tuple to update scan key. It could be at
+ * the end, so check for max offset
+ */
+ if (!_bt_readpage(scan, ForwardScanDirection, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ LockBuffer(so->currPos.buf, BT_READ);
+ }
+
+ currItem = &so->currPos.items[so->currPos.firstItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ scan->xs_itup = itup;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+
+ _bt_update_skip_scankeys(scan, indexRel);
+ if (BufferIsValid(so->currPos.buf))
+ {
+ /* Before leaving current page, deal with any killed items */
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ jumpOffset = offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ offnum = OffsetNumberPrev(offnum);
+
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, indexdir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ LockBuffer(so->currPos.buf, BT_READ);
+ }
+
+ currItem = &so->currPos.items[so->currPos.lastItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ nextOffset = ItemPointerGetOffsetNumber(&itup->t_tid);
+
+ /*
+ * To check if we returned the same tuple, try to find a
+ * startItup on the current page. For that we need to update
+ * scankey to match the whole tuple and set nextkey to return
+ * an exact tuple, not the next one. If the nextOffset is the
+ * same as before, it means we are in the loop, return offnum
+ * to the original position and jump further
+ */
+ scan->xs_itup = startItup;
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ so->skipScanKey->keysz = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ so->skipScanKey->nextkey = false;
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey,
+ so->currPos.buf, dir))
+ {
+ OffsetNumber maxoff;
+ startOffset = _bt_binsrch(scan->indexRelation,
+ so->skipScanKey,
+ so->currPos.buf);
+
+ page = BufferGetPage(so->currPos.buf);
+ maxoff = PageGetMaxOffsetNumber(page);
+
+ if (nextOffset <= startOffset)
+ {
+ offnum = jumpOffset;
+ nextOffset = startOffset;
+ }
+
+ if ((offnum > maxoff) & (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+
+ /* Return original scankey options */
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ }
+ }
+ }
+
+ /* Now read the data */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to advance to
+ * the next page. Return false if there's no matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ else
+ {
+ /* Drop the lock, and maybe the pin, on the current page */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ /* And set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ return true;
+}
+
/*
* _bt_readpage() -- Load data from current index page into so->currPos
*
@@ -2246,3 +2659,54 @@ _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
so->numKilled = 0; /* just paranoia */
so->markItemIndex = -1; /* ditto */
}
+
+/*
+ * _bt_update_skip_scankeys() -- set up a new values for the existing scankeys
+ * based on the current index tuple
+ */
+static inline void
+_bt_update_skip_scankeys(IndexScanDesc scan, Relation indexRel)
+{
+ TupleDesc itupdesc;
+ int indnkeyatts,
+ i;
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ ScanKey scankeys = so->skipScanKey->scankeys;
+
+ itupdesc = RelationGetDescr(indexRel);
+ indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ for (i = 0; i < indnkeyatts; i++)
+ {
+ Datum datum;
+ bool null;
+ int flags;
+
+ datum = index_getattr(scan->xs_itup, i + 1, itupdesc, &null);
+ flags = (null ? SK_ISNULL : 0) |
+ (indexRel->rd_indoption[i] << SK_BT_INDOPTION_SHIFT);
+ scankeys[i].sk_flags = flags;
+ scankeys[i].sk_argument = datum;
+ }
+}
+
+/*
+ * _bt_scankey_within_page() -- check if the provided scankey could be found
+ * within a page, specified by the buffer.
+ */
+static inline bool
+_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir)
+{
+ OffsetNumber low, high;
+ Page page = BufferGetPage(buf);
+ BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ low = P_FIRSTDATAKEY(opaque);
+ high = PageGetMaxOffsetNumber(page);
+
+ if (unlikely(high < low))
+ return false;
+
+ return (_bt_compare(scan->indexRelation, key, page, low) > 0 &&
+ _bt_compare(scan->indexRelation, key, page, high) < 1);
+}
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4924ae1c59..fa09a4685e 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -68,6 +68,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index c367c750b1..a7dd874531 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -141,6 +141,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1052,6 +1053,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1386,6 +1403,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1396,6 +1415,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1655,6 +1676,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1668,6 +1691,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 54ad62bb7f..e0cfd710c4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 16083e7a7e..5f723cda4b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 551ce6c41c..028d03a56d 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1820,6 +1820,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1839,6 +1840,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index b5a0033721..710edf160a 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -124,6 +124,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index dff826a828..7b32f2cc7e 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -175,12 +175,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2910,7 +2912,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2921,7 +2924,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5184,7 +5188,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5201,6 +5206,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5213,7 +5219,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5228,6 +5235,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 984fca0696..c84388e6f7 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4834,6 +4834,82 @@ create_distinct_paths(PlannerInfo *root,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+
+ /* Consider index skip scan as well */
+ if (enable_indexskipscan &&
+ IsA(path, IndexPath) &&
+ ((IndexPath *) path)->indexinfo->amcanskip &&
+ root->distinct_pathkeys != NIL)
+ {
+ ListCell *lc;
+ IndexOptInfo *index = NULL;
+ bool different_columns_order = false,
+ not_empty_qual = false;
+ int i = 0;
+ int distinctPrefixKeys;
+
+ Assert(path->pathtype == T_IndexOnlyScan ||
+ path->pathtype == T_IndexScan);
+
+ index = ((IndexPath *) path)->indexinfo;
+ distinctPrefixKeys = list_length(root->query_uniquekeys);
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniquekey = (UniqueKey *) lfirst(lc);
+ EquivalenceMember *em =
+ lfirst_node(EquivalenceMember,
+ list_head(uniquekey->eq_clause->ec_members));
+ Var *var = (Var *) em->em_expr;
+
+ Assert(i < index->ncolumns);
+
+ for (i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ /*
+ * XXX: In case of index scan quals evaluation happens
+ * after ExecScanFetch, which means skip results could be
+ * fitered out. Consider the following query:
+ *
+ * select distinct (a, b) a, b, c from t where c < 100;
+ *
+ * Skip scan returns one tuple for one distinct set of (a,
+ * b) with arbitrary one of c, so if the choosed c does
+ * not match the qual and there is any c that matches the
+ * qual, we miss that tuple.
+ */
+ if (path->pathtype == T_IndexScan &&
+ parse->jointree != NULL &&
+ parse->jointree->quals != NULL &&
+ list_length((List *) parse->jointree->quals) != 0)
+ not_empty_qual = true;
+
+ if (!different_columns_order && !not_empty_qual)
+ {
+ add_path(distinct_rel, (Path *)
+ create_skipscan_unique_path(root,
+ distinct_rel,
+ path,
+ distinctPrefixKeys,
+ numDistinctRows));
+ }
+ }
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index a006dbbe9c..d483ec38f2 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2915,6 +2915,45 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *basepath,
+ int distinctPrefixKeys,
+ double numGroups)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /* The size of the prefix we'll use for skipping. */
+ Assert(pathnode->indexinfo->amcanskip);
+ Assert(distinctPrefixKeys > 0);
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ /*
+ * The cost to skip to each distinct value should be roughly the same as
+ * the cost of finding the first key times the number of distinct values
+ * we expect to find.
+ */
+ pathnode->path.startup_cost = basepath->startup_cost;
+ pathnode->path.total_cost = basepath->startup_cost * numGroups;
+ pathnode->path.rows = numGroups;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index d82fc5ab8b..f65b299f37 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -271,6 +271,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index cacbe904db..7c71ee4499 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -923,6 +923,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e1048c0047..a002ee2143 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -353,6 +353,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 3b3e22f73d..3d39cd9d07 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -229,6 +236,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 7e9364a50c..815de4e4dd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 20ace69dab..e098c6a1ab 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -662,6 +662,9 @@ typedef struct BTScanOpaqueData
*/
int markItemIndex; /* itemIndex, or -1 if not valid */
+ /* Work space for _bt_skip */
+ BTScanInsert skipScanKey; /* used to control skipping */
+
/* keep these last in struct for efficiency */
BTScanPosData currPos; /* current position data */
BTScanPosData markPos; /* marked position, if any */
@@ -793,6 +796,8 @@ extern OffsetNumber _bt_binsrch_insert(Relation rel, BTInsertState insertstate);
extern int32 _bt_compare(Relation rel, BTScanInsert key, Page page, OffsetNumber offnum);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
+extern bool _bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
Snapshot snapshot);
@@ -817,6 +822,8 @@ extern void _bt_end_vacuum_callback(int code, Datum arg);
extern Size BTreeShmemSize(void);
extern void BTreeShmemInit(void);
extern bytea *btoptions(Datum reloptions, bool validate);
+extern bool btskip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 1f6f5bbc20..2c6acc160a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1423,6 +1423,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1452,6 +1454,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1470,6 +1474,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 4e329f0fb5..b0ff9ca3a8 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -839,6 +839,7 @@ struct IndexOptInfo
bool amsearchnulls; /* can AM search for NULL/NOT NULL entries? */
bool amhasgettuple; /* does AM have amgettuple interface? */
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
+ bool amcanskip; /* can AM skip duplicate values? */
bool amcanparallel; /* does AM support parallel scan? */
/* Rather than include amapi.h here, we declare amcostestimate like this */
void (*amcostestimate) (); /* AM's cost estimator */
@@ -1189,6 +1190,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1201,6 +1205,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 32c0d87f80..03a00e8e1d 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index cb012ba198..847f34f02b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index f75ff6f323..6c8c9dadbb 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -201,6 +201,11 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *subpath,
+ int numCols,
+ double numGroups);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/test/regress/expected/select_distinct.out b/src/test/regress/expected/select_distinct.out
index f3696c6d1d..259db10c81 100644
--- a/src/test/regress/expected/select_distinct.out
+++ b/src/test/regress/expected/select_distinct.out
@@ -244,3 +244,604 @@ SELECT null IS NOT DISTINCT FROM null as "yes";
t
(1 row)
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+SELECT DISTINCT a FROM distinct_a;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+ a
+---
+ 1
+(1 row)
+
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+ QUERY PLAN
+--------------------------------------------------------
+ Index Only Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+DROP INDEX distinct_a_b_a;
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+FETCH FROM c;
+ a | b
+---+---
+ 1 | 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+END;
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+FETCH FROM c;
+ a | b
+---+-------
+ 5 | 10000
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+END;
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+ QUERY PLAN
+--------------------------------------------------------------
+ Index Only Scan using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 1 | 2
+ 3 | 1 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 1 | 2
+ 1 | 1 | 2
+(2 rows)
+
+END;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+ QUERY PLAN
+-----------------------------------------------------------------------
+ Index Only Scan Backward using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 2 | 2
+ 1 | 2 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 2 | 2
+ 3 | 2 | 2
+(2 rows)
+
+END;
+DROP TABLE distinct_abc;
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+ 2 | 1 | 10
+ 3 | 1 | 10
+ 4 | 1 | 10
+ 5 | 1 | 10
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ QUERY PLAN
+---------------------------------------------------
+ Index Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ QUERY PLAN
+-----------------------------------------------------
+ Unique
+ -> Bitmap Heap Scan on distinct_a
+ Recheck Cond: (a = 1)
+ -> Bitmap Index Scan on distinct_a_a_b_idx
+ Index Cond: (a = 1)
+(5 rows)
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ QUERY PLAN
+---------------------------------------------------------
+ Unique
+ -> Index Scan using distinct_a_a_b_idx on distinct_a
+ Index Cond: (b = 2)
+ Filter: (c = 10)
+(4 rows)
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+ a | a
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 3
+ 4 | 4
+ 5 | 5
+(5 rows)
+
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+ a | ?column?
+---+----------
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+FETCH FROM c;
+ a
+---
+ 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a
+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+END;
+DROP TABLE distinct_a;
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 9999
+ 1 | 10000
+(5 rows)
+
+DROP TABLE distinct_visibility;
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Index Only Scan using distinct_boundaries_a_b_c_idx on distinct_boundaries
+ Skip scan: true
+ Index Cond: ((b >= 1) AND (c = 0))
+(3 rows)
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ a | b | c
+---+---+---
+ 1 | 2 | 0
+ 2 | 2 | 0
+ 3 | 2 | 0
+ 4 | 2 | 0
+ 5 | 2 | 0
+(5 rows)
+
+DROP TABLE distinct_boundaries;
+-- test tuple killing
+-- DESC ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed where a = 3;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 5 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 1 | 1000 | 0 | 10
+(4 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 1 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 5 | 1000 | 0 | 10
+(4 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
+-- regular ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed where a = 3;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a, b;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+---+---+----
+ 1 | 1 | 1 | 10
+ 2 | 1 | 1 | 10
+ 4 | 1 | 1 | 10
+ 5 | 1 | 1 | 10
+(4 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+---+---+----
+ 5 | 1 | 1 | 10
+ 4 | 1 | 1 | 10
+ 2 | 1 | 1 | 10
+ 1 | 1 | 1 | 10
+(4 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
+-- partial delete
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed WHERE a = 3 AND b <= 999;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 5 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 3 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 1 | 1000 | 0 | 10
+(5 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 1 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 3 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 5 | 1000 | 0 | 10
+(5 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a1c90eb905..bd3b373515 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -78,6 +78,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_hashjoin | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -89,7 +90,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(17 rows)
+(18 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/sql/select_distinct.sql b/src/test/regress/sql/select_distinct.sql
index a605e86449..843efeb28f 100644
--- a/src/test/regress/sql/select_distinct.sql
+++ b/src/test/regress/sql/select_distinct.sql
@@ -73,3 +73,251 @@ SELECT 1 IS NOT DISTINCT FROM 2 as "no";
SELECT 2 IS NOT DISTINCT FROM 2 as "yes";
SELECT 2 IS NOT DISTINCT FROM null as "no";
SELECT null IS NOT DISTINCT FROM null as "yes";
+
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+
+SELECT DISTINCT a FROM distinct_a;
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+DROP INDEX distinct_a_b_a;
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+DROP TABLE distinct_abc;
+
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+DROP TABLE distinct_a;
+
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DROP TABLE distinct_visibility;
+
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+DROP TABLE distinct_boundaries;
+
+-- test tuple killing
+
+-- DESC ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed where a = 3;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
+
+-- regular ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed where a = 3;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a, b;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
+
+-- partial delete
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed WHERE a = 3 AND b <= 999;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
--
2.21.0
--f7z2si56qzkbcsr7--
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v33 2/2] Index skip scan
@ 2019-11-15 14:46 jesperpedersen <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: jesperpedersen @ 2019-11-15 14:46 UTC (permalink / raw)
Implementation of Index Skip Scan (see Loose Index Scan in the wiki [1])
on top of IndexOnlyScan and IndexScan. To make it suitable for both
situations when there are small number of distinct values and
significant amount of distinct values the following approach is taken -
instead of searching from the root for every value we're searching for
then first on the current page, and then if not found continue searching
from the root.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 +
doc/src/sgml/indexam.sgml | 63 ++
doc/src/sgml/indices.sgml | 23 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 +
src/backend/access/nbtree/nbtree.c | 13 +
src/backend/access/nbtree/nbtsearch.c | 469 ++++++++++++-
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +
src/backend/executor/nodeIndexonlyscan.c | 97 ++-
src/backend/executor/nodeIndexscan.c | 56 +-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 78 +++
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/planner.c | 10 +-
src/backend/optimizer/util/pathnode.c | 68 ++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/nbtree.h | 7 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 5 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 3 +
src/test/regress/expected/select_distinct.out | 621 ++++++++++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/select_distinct.sql | 254 +++++++
38 files changed, 1886 insertions(+), 14 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 0104d02f67..a018b7f3d0 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -133,6 +133,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e07dc01e80..36ba75b077 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4517,6 +4517,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 37f8d8760a..a726d80878 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -148,6 +148,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -691,6 +692,68 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ ScanDirection indexdir,
+ bool scanstart,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>indexdir</parameter></term>
+ <listitem>
+ <para>
+ The index direction, in which data must be read.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>scanstart</parameter></term>
+ <listitem>
+ <para>
+ Whether or not it is a start of the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index c54bf0dbbd..c429d98fc7 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1254,6 +1254,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 2e8f67ef10..4db31bb211 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -113,6 +113,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a7e55caf28..8dd1d30d2a 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -65,6 +65,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index aefc302ed2..8c692f7fb4 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 4871b7ff4d..e5fa4c7864 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 01539b6bd6..1047a35ade 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -730,6 +731,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 5254bc7ef5..8fde56fe60 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -132,6 +132,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
+ amroutine->amskip = btskip;
amroutine->amcostestimate = btcostestimate;
amroutine->amoptions = btoptions;
amroutine->amproperty = btproperty;
@@ -381,6 +382,8 @@ btbeginscan(Relation rel, int nkeys, int norderbys)
*/
so->currTuples = so->markTuples = NULL;
+ so->skipScanKey = NULL;
+
scan->xs_itupdesc = RelationGetDescr(rel);
scan->opaque = so;
@@ -448,6 +451,16 @@ btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
_bt_preprocess_array_keys(scan);
}
+/*
+ * btskip() -- skip to the beginning of the next key prefix
+ */
+bool
+btskip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix)
+{
+ return _bt_skip(scan, direction, indexdir, start, prefix);
+}
+
/*
* btendscan() -- close down a scan
*/
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index c573814f01..e2b549355b 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -37,7 +37,10 @@ static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
static Buffer _bt_walk_left(Relation rel, Buffer buf, Snapshot snapshot);
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
-
+static inline void _bt_update_skip_scankeys(IndexScanDesc scan,
+ Relation indexRel);
+static inline bool _bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir);
/*
* _bt_drop_lock_and_maybe_pin()
@@ -1375,6 +1378,419 @@ _bt_next(IndexScanDesc scan, ScanDirection dir)
return true;
}
+/*
+ * _bt_skip() -- Skip items that have the same prefix as the most recently
+ * fetched index tuple.
+ *
+ * The current position is set so that a subsequent call to _bt_next will
+ * fetch the first tuple that differs in the leading 'prefix' keys.
+ *
+ * There are four different kinds of skipping (depending on dir and
+ * indexdir, that are important to distinguish, especially in the presense
+ * of an index condition:
+ *
+ * * Advancing forward and reading forward
+ * simple scan
+ *
+ * * Advancing forward and reading backward
+ * scan inside a cursor fetching backward, when skipping is necessary
+ * right from the start
+ *
+ * * Advancing backward and reading forward
+ * scan with order by desc inside a cursor fetching forward, when
+ * skipping is necessary right from the start
+ *
+ * * Advancing backward and reading backward
+ * simple scan with order by desc
+ *
+ * The current page is searched for the next unique value. If none is found
+ * we will do a scan from the root in order to find the next page with
+ * a unique value.
+ */
+bool
+_bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ BTStack stack;
+ Buffer buf;
+ OffsetNumber offnum;
+ BTScanPosItem *currItem;
+ Relation indexRel = scan->indexRelation;
+
+ /* We want to return tuples, and we need a starting point */
+ Assert(scan->xs_want_itup);
+ Assert(scan->xs_itup);
+
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ /* If skipScanKey is NULL then we initialize it with _bt_mkscankey */
+ if (so->skipScanKey == NULL)
+ {
+ so->skipScanKey = _bt_mkscankey(indexRel, scan->xs_itup);
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->scantid = NULL;
+ }
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /* Check if the next unique key can be found within the current page.
+ * Since we do not lock the current page between jumps, it's possible
+ * that it was splitted since the last time we saw it. This is fine in
+ * case of scanning forward, since page split to the right and we are
+ * still on the left most page. In case of scanning backwards it's
+ * possible to loose some pages and we need to remember the previous
+ * page, and then follow the right link from the current page until we
+ * find the original one.
+ *
+ * Since the whole idea of checking the current page is to protect
+ * ourselves and make more performant statistic mismatch case when
+ * there are too many distinct values for jumping, it's not clear if
+ * the complexity of this solution in case of backward scan is
+ * justified, so for now just avoid it.
+ */
+ if (BufferIsValid(so->currPos.buf) && ScanDirectionIsForward(dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ {
+ bool keyFound = false;
+
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, so->currPos.buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(so->currPos.buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /* Now read the data */
+ keyFound = _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+
+ if (keyFound)
+ {
+ /* set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+ }
+ }
+ else
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+ }
+
+ if (BufferIsValid(so->currPos.buf))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ /*
+ * We haven't found scan key within the current page, so let's scan from
+ * the root. Use _bt_search and _bt_binsrch to get the buffer and offset
+ * number
+ */
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /*
+ * Simplest case is when both directions are forward, when we are already
+ * at the next distinct key at the beginning of the series (so everything
+ * else would be done in _bt_readpage)
+ *
+ * The case when both directions are backwards is also simple, but we need
+ * to go one step back, since we need a last element from the previous
+ * series.
+ */
+ if (ScanDirectionIsBackward(dir) && ScanDirectionIsBackward(indexdir))
+ offnum = OffsetNumberPrev(offnum);
+
+ /*
+ * Andvance backward but read forward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can read forward without doing anything else. Otherwise
+ * find previous distinct key and the beginning of it's series and read
+ * forward from there. To do so, go back one step, perform binary search
+ * to find the first item in the series and let _bt_readpage do everything
+ * else.
+ */
+ else if (ScanDirectionIsBackward(dir) && ScanDirectionIsForward(indexdir))
+ {
+ if (!scanstart)
+ {
+ /* Reading forward means we expect to see more data on the right */
+ so->currPos.moreRight = true;
+
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* One step back to find a previous value */
+ _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (_bt_next(scan, dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /*
+ * And now find the last item from the sequence for the
+ * current, value with the intention do OffsetNumberNext. As a
+ * result we end up on a first element from the sequence.
+ */
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ else
+ {
+ if (BufferIsValid(so->currPos.buf))
+ {
+ /* Before leaving current page, deal with any killed items */
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ }
+ }
+ else
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ }
+
+ /*
+ * Advance forward but read backward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can go one step back and read forward without doing
+ * anything else. Otherwise find the next distinct key and the beginning
+ * of it's series, go one step back and read backward from there.
+ *
+ * An interesting situation can happen if one of distinct keys do not pass
+ * a corresponding index condition at all. In this case reading backward
+ * can lead to a previous distinct key being found, creating a loop. To
+ * avoid that check the value to be returned, and jump one more time if
+ * it's the same as at the beginning. Note that we do not check visibility
+ * here, and dead tuples could also lead to the same situation. This has to
+ * be checked on the caller side.
+ */
+ else if (ScanDirectionIsForward(dir) && ScanDirectionIsBackward(indexdir))
+ {
+ if (scanstart)
+ offnum = OffsetNumberPrev(offnum);
+ else
+ {
+ OffsetNumber nextOffset,
+ startOffset,
+ jumpOffset;
+
+ IndexTuple startItup = CopyIndexTuple(scan->xs_itup);
+ Page page = BufferGetPage(so->currPos.buf);
+
+ /* We are at the end and need to return */
+ if ((offnum > PageGetMaxOffsetNumber(page)) &
+ (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+
+ nextOffset = startOffset = ItemPointerGetOffsetNumber(&scan->xs_itup->t_tid);
+
+ /* Reading backwards means we expect to see more data on the left */
+ so->currPos.moreLeft = true;
+
+ while (nextOffset == startOffset)
+ {
+ IndexTuple itup;
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * Find a next index tuple to update scan key. It could be at
+ * the end, so check for max offset
+ */
+ if (!_bt_readpage(scan, ForwardScanDirection, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ LockBuffer(so->currPos.buf, BT_READ);
+ }
+
+ currItem = &so->currPos.items[so->currPos.firstItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ scan->xs_itup = itup;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+
+ _bt_update_skip_scankeys(scan, indexRel);
+ if (BufferIsValid(so->currPos.buf))
+ {
+ /* Before leaving current page, deal with any killed items */
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ jumpOffset = offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ offnum = OffsetNumberPrev(offnum);
+
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, indexdir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ LockBuffer(so->currPos.buf, BT_READ);
+ }
+
+ currItem = &so->currPos.items[so->currPos.lastItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ nextOffset = ItemPointerGetOffsetNumber(&itup->t_tid);
+
+ /*
+ * To check if we returned the same tuple, try to find a
+ * startItup on the current page. For that we need to update
+ * scankey to match the whole tuple and set nextkey to return
+ * an exact tuple, not the next one. If the nextOffset is the
+ * same as before, it means we are in the loop, return offnum
+ * to the original position and jump further
+ */
+ scan->xs_itup = startItup;
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ so->skipScanKey->keysz = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ so->skipScanKey->nextkey = false;
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey,
+ so->currPos.buf, dir))
+ {
+ OffsetNumber maxoff;
+ startOffset = _bt_binsrch(scan->indexRelation,
+ so->skipScanKey,
+ so->currPos.buf);
+
+ page = BufferGetPage(so->currPos.buf);
+ maxoff = PageGetMaxOffsetNumber(page);
+
+ if (nextOffset <= startOffset)
+ {
+ offnum = jumpOffset;
+ nextOffset = startOffset;
+ }
+
+ if ((offnum > maxoff) & (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+
+ /* Return original scankey options */
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ }
+ }
+ }
+
+ /* Now read the data */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to advance to
+ * the next page. Return false if there's no matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ else
+ {
+ /* Drop the lock, and maybe the pin, on the current page */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ /* And set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ so->currPos.moreLeft = true;
+ so->currPos.moreRight = true;
+
+ return true;
+}
+
/*
* _bt_readpage() -- Load data from current index page into so->currPos
*
@@ -2246,3 +2662,54 @@ _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
so->numKilled = 0; /* just paranoia */
so->markItemIndex = -1; /* ditto */
}
+
+/*
+ * _bt_update_skip_scankeys() -- set up a new values for the existing scankeys
+ * based on the current index tuple
+ */
+static inline void
+_bt_update_skip_scankeys(IndexScanDesc scan, Relation indexRel)
+{
+ TupleDesc itupdesc;
+ int indnkeyatts,
+ i;
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ ScanKey scankeys = so->skipScanKey->scankeys;
+
+ itupdesc = RelationGetDescr(indexRel);
+ indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ for (i = 0; i < indnkeyatts; i++)
+ {
+ Datum datum;
+ bool null;
+ int flags;
+
+ datum = index_getattr(scan->xs_itup, i + 1, itupdesc, &null);
+ flags = (null ? SK_ISNULL : 0) |
+ (indexRel->rd_indoption[i] << SK_BT_INDOPTION_SHIFT);
+ scankeys[i].sk_flags = flags;
+ scankeys[i].sk_argument = datum;
+ }
+}
+
+/*
+ * _bt_scankey_within_page() -- check if the provided scankey could be found
+ * within a page, specified by the buffer.
+ */
+static inline bool
+_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir)
+{
+ OffsetNumber low, high;
+ Page page = BufferGetPage(buf);
+ BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ low = P_FIRSTDATAKEY(opaque);
+ high = PageGetMaxOffsetNumber(page);
+
+ if (unlikely(high < low))
+ return false;
+
+ return (_bt_compare(scan->indexRelation, key, page, low) > 0 &&
+ _bt_compare(scan->indexRelation, key, page, high) < 1);
+}
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4924ae1c59..fa09a4685e 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -68,6 +68,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index c367c750b1..a7dd874531 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -141,6 +141,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1052,6 +1053,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1386,6 +1403,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1396,6 +1415,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1655,6 +1676,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1668,6 +1691,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 54ad62bb7f..e0cfd710c4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 16083e7a7e..5f723cda4b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 551ce6c41c..028d03a56d 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1820,6 +1820,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1839,6 +1840,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index b5a0033721..710edf160a 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -124,6 +124,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 363f5349f1..196b132568 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -791,6 +791,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -880,6 +890,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -1029,6 +1041,60 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1056,6 +1122,12 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (useful_uniquekeys != NULL && can_skip && !not_empty_qual)
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath));
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1116,6 +1188,12 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (useful_uniquekeys != NULL && can_skip && !not_empty_qual)
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath));
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index dff826a828..7b32f2cc7e 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -175,12 +175,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2910,7 +2912,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2921,7 +2924,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5184,7 +5188,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5201,6 +5206,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5213,7 +5219,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5228,6 +5235,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index a7de8476d9..88305df5c3 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4828,13 +4828,19 @@ create_distinct_paths(PlannerInfo *root,
Path *path = (Path *) lfirst(lc);
if (pathkeys_contained_in(needed_pathkeys, path->pathkeys))
- {
add_path(distinct_rel, (Path *)
create_upper_unique_path(root, distinct_rel,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
- }
+ }
+
+ foreach(lc, input_rel->unique_pathlist)
+ {
+ Path *path = (Path *) lfirst(lc);
+
+ if (uniquekeys_contained_in(needed_pathkeys, path->uniquekeys))
+ add_path(distinct_rel, path);
}
/* For explicit-sort case, always use the more rigorous clause */
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index a4dfafbb59..b0ce17b0d6 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2564,6 +2564,7 @@ create_projection_path(PlannerInfo *root,
pathnode->path.pathkeys = subpath->pathkeys;
pathnode->subpath = subpath;
+ pathnode->path.uniquekeys = subpath->uniquekeys;
/*
* We might not need a separate Result node. If the input plan node type
@@ -2929,6 +2930,73 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ int distinctPrefixKeys;
+ ListCell *lc;
+ List *exprs = NIL;
+
+
+ distinctPrefixKeys = list_length(root->query_uniquekeys);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniquekey = (UniqueKey *) lfirst(lc);
+ EquivalenceMember *em =
+ lfirst_node(EquivalenceMember,
+ list_head(uniquekey->eq_clause->ec_members));
+ Var *var = (Var *) em->em_expr;
+
+ exprs = lappend(exprs, em->em_expr);
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ Assert(distinctPrefixKeys > 0);
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ numDistinctRows = estimate_num_groups(root, exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index d82fc5ab8b..f65b299f37 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -271,6 +271,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index cacbe904db..7c71ee4499 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -923,6 +923,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e1048c0047..a002ee2143 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -353,6 +353,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 3b3e22f73d..3d39cd9d07 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -229,6 +236,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 7e9364a50c..815de4e4dd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 20ace69dab..e098c6a1ab 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -662,6 +662,9 @@ typedef struct BTScanOpaqueData
*/
int markItemIndex; /* itemIndex, or -1 if not valid */
+ /* Work space for _bt_skip */
+ BTScanInsert skipScanKey; /* used to control skipping */
+
/* keep these last in struct for efficiency */
BTScanPosData currPos; /* current position data */
BTScanPosData markPos; /* marked position, if any */
@@ -793,6 +796,8 @@ extern OffsetNumber _bt_binsrch_insert(Relation rel, BTInsertState insertstate);
extern int32 _bt_compare(Relation rel, BTScanInsert key, Page page, OffsetNumber offnum);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
+extern bool _bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
Snapshot snapshot);
@@ -817,6 +822,8 @@ extern void _bt_end_vacuum_callback(int code, Datum arg);
extern Size BTreeShmemSize(void);
extern void BTreeShmemInit(void);
extern bytea *btoptions(Datum reloptions, bool validate);
+extern bool btskip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 1f6f5bbc20..2c6acc160a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1423,6 +1423,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1452,6 +1454,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1470,6 +1474,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 0de27f0ef3..ce00060ee0 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -840,6 +840,7 @@ struct IndexOptInfo
bool amsearchnulls; /* can AM search for NULL/NOT NULL entries? */
bool amhasgettuple; /* does AM have amgettuple interface? */
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
+ bool amcanskip; /* can AM skip duplicate values? */
bool amcanparallel; /* does AM support parallel scan? */
/* Rather than include amapi.h here, we declare amcostestimate like this */
void (*amcostestimate) (); /* AM's cost estimator */
@@ -1190,6 +1191,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1202,6 +1206,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 32c0d87f80..03a00e8e1d 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index cb012ba198..847f34f02b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index fd25997af5..ba3eaffd8a 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -202,6 +202,9 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/test/regress/expected/select_distinct.out b/src/test/regress/expected/select_distinct.out
index f3696c6d1d..c50c6d1866 100644
--- a/src/test/regress/expected/select_distinct.out
+++ b/src/test/regress/expected/select_distinct.out
@@ -244,3 +244,624 @@ SELECT null IS NOT DISTINCT FROM null as "yes";
t
(1 row)
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+CREATE INDEX ON distinct_a ((a + 1));
+ANALYZE distinct_a;
+SELECT DISTINCT a FROM distinct_a;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+ a
+---
+ 1
+(1 row)
+
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+ QUERY PLAN
+--------------------------------------------------------
+ Index Only Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+-- test index skip scan for expressions
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT (a + 1) FROM distinct_a;
+ QUERY PLAN
+----------------------------------------------------
+ Index Scan using distinct_a_expr_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+SELECT DISTINCT (a + 1) FROM distinct_a;
+ ?column?
+----------
+ 2
+ 3
+ 4
+ 5
+ 6
+(5 rows)
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+DROP INDEX distinct_a_b_a;
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+FETCH FROM c;
+ a | b
+---+---
+ 1 | 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+END;
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+FETCH FROM c;
+ a | b
+---+-------
+ 5 | 10000
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+END;
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+ QUERY PLAN
+--------------------------------------------------------------
+ Index Only Scan using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 1 | 2
+ 3 | 1 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 1 | 2
+ 1 | 1 | 2
+(2 rows)
+
+END;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+ QUERY PLAN
+-----------------------------------------------------------------------
+ Index Only Scan Backward using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 2 | 2
+ 1 | 2 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 2 | 2
+ 3 | 2 | 2
+(2 rows)
+
+END;
+DROP TABLE distinct_abc;
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+ 2 | 1 | 10
+ 3 | 1 | 10
+ 4 | 1 | 10
+ 5 | 1 | 10
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ QUERY PLAN
+---------------------------------------------------
+ Index Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ QUERY PLAN
+-----------------------------------------------------
+ Unique
+ -> Bitmap Heap Scan on distinct_a
+ Recheck Cond: (a = 1)
+ -> Bitmap Index Scan on distinct_a_a_b_idx
+ Index Cond: (a = 1)
+(5 rows)
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ QUERY PLAN
+---------------------------------------------------------
+ Unique
+ -> Index Scan using distinct_a_a_b_idx on distinct_a
+ Index Cond: (b = 2)
+ Filter: (c = 10)
+(4 rows)
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+ a | a
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 3
+ 4 | 4
+ 5 | 5
+(5 rows)
+
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+ a | ?column?
+---+----------
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+FETCH FROM c;
+ a
+---
+ 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a
+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+END;
+DROP TABLE distinct_a;
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 9999
+ 1 | 10000
+(5 rows)
+
+DROP TABLE distinct_visibility;
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Index Only Scan using distinct_boundaries_a_b_c_idx on distinct_boundaries
+ Skip scan: true
+ Index Cond: ((b >= 1) AND (c = 0))
+(3 rows)
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ a | b | c
+---+---+---
+ 1 | 2 | 0
+ 2 | 2 | 0
+ 3 | 2 | 0
+ 4 | 2 | 0
+ 5 | 2 | 0
+(5 rows)
+
+DROP TABLE distinct_boundaries;
+-- test tuple killing
+-- DESC ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed where a = 3;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 5 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 1 | 1000 | 0 | 10
+(4 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 1 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 5 | 1000 | 0 | 10
+(4 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
+-- regular ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed where a = 3;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a, b;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+---+---+----
+ 1 | 1 | 1 | 10
+ 2 | 1 | 1 | 10
+ 4 | 1 | 1 | 10
+ 5 | 1 | 1 | 10
+(4 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+---+---+----
+ 5 | 1 | 1 | 10
+ 4 | 1 | 1 | 10
+ 2 | 1 | 1 | 10
+ 1 | 1 | 1 | 10
+(4 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
+-- partial delete
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed WHERE a = 3 AND b <= 999;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 5 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 3 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 1 | 1000 | 0 | 10
+(5 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 1 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 3 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 5 | 1000 | 0 | 10
+(5 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a1c90eb905..bd3b373515 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -78,6 +78,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_hashjoin | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -89,7 +90,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(17 rows)
+(18 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/sql/select_distinct.sql b/src/test/regress/sql/select_distinct.sql
index a605e86449..3441a0efc6 100644
--- a/src/test/regress/sql/select_distinct.sql
+++ b/src/test/regress/sql/select_distinct.sql
@@ -73,3 +73,257 @@ SELECT 1 IS NOT DISTINCT FROM 2 as "no";
SELECT 2 IS NOT DISTINCT FROM 2 as "yes";
SELECT 2 IS NOT DISTINCT FROM null as "no";
SELECT null IS NOT DISTINCT FROM null as "yes";
+
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+CREATE INDEX ON distinct_a ((a + 1));
+ANALYZE distinct_a;
+
+SELECT DISTINCT a FROM distinct_a;
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+
+-- test index skip scan for expressions
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT (a + 1) FROM distinct_a;
+SELECT DISTINCT (a + 1) FROM distinct_a;
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+DROP INDEX distinct_a_b_a;
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+DROP TABLE distinct_abc;
+
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+DROP TABLE distinct_a;
+
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DROP TABLE distinct_visibility;
+
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+DROP TABLE distinct_boundaries;
+
+-- test tuple killing
+
+-- DESC ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed where a = 3;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
+
+-- regular ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed where a = 3;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a, b;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
+
+-- partial delete
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed WHERE a = 3 AND b <= 999;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
--
2.21.0
--t7fko7qta7zyrtxo--
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v31 2/2] Index skip scan
@ 2019-11-15 14:46 jesperpedersen <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: jesperpedersen @ 2019-11-15 14:46 UTC (permalink / raw)
Implementation of Index Skip Scan (see Loose Index Scan in the wiki [1])
on top of IndexOnlyScan and IndexScan. To make it suitable for both
situations when there are small number of distinct values and
significant amount of distinct values the following approach is taken -
instead of searching from the root for every value we're searching for
then first on the current page, and then if not found continue searching
from the root.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 +
doc/src/sgml/indexam.sgml | 63 +++
doc/src/sgml/indices.sgml | 23 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 +
src/backend/access/nbtree/nbtree.c | 13 +
src/backend/access/nbtree/nbtsearch.c | 425 +++++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +
src/backend/executor/nodeIndexonlyscan.c | 51 +-
src/backend/executor/nodeIndexscan.c | 56 +-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/planner.c | 76 +++
src/backend/optimizer/util/pathnode.c | 40 ++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/nbtree.h | 7 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 5 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 5 +
src/test/regress/expected/select_distinct.out | 505 ++++++++++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/select_distinct.sql | 186 +++++++
37 files changed, 1577 insertions(+), 11 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 0104d02f67..a018b7f3d0 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -133,6 +133,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d45b6f7cb..f15bc20c20 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4517,6 +4517,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 37f8d8760a..a726d80878 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -148,6 +148,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -691,6 +692,68 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ ScanDirection indexdir,
+ bool scanstart,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>indexdir</parameter></term>
+ <listitem>
+ <para>
+ The index direction, in which data must be read.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>scanstart</parameter></term>
+ <listitem>
+ <para>
+ Whether or not it is a start of the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index c54bf0dbbd..c429d98fc7 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1254,6 +1254,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 2e8f67ef10..4db31bb211 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -113,6 +113,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a7e55caf28..8dd1d30d2a 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -65,6 +65,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index aefc302ed2..8c692f7fb4 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 4871b7ff4d..e5fa4c7864 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 01539b6bd6..1047a35ade 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -730,6 +731,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 5254bc7ef5..8fde56fe60 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -132,6 +132,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
+ amroutine->amskip = btskip;
amroutine->amcostestimate = btcostestimate;
amroutine->amoptions = btoptions;
amroutine->amproperty = btproperty;
@@ -381,6 +382,8 @@ btbeginscan(Relation rel, int nkeys, int norderbys)
*/
so->currTuples = so->markTuples = NULL;
+ so->skipScanKey = NULL;
+
scan->xs_itupdesc = RelationGetDescr(rel);
scan->opaque = so;
@@ -448,6 +451,16 @@ btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
_bt_preprocess_array_keys(scan);
}
+/*
+ * btskip() -- skip to the beginning of the next key prefix
+ */
+bool
+btskip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix)
+{
+ return _bt_skip(scan, direction, indexdir, start, prefix);
+}
+
/*
* btendscan() -- close down a scan
*/
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index c573814f01..c5f5d228f2 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -37,6 +37,10 @@ static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
static Buffer _bt_walk_left(Relation rel, Buffer buf, Snapshot snapshot);
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
+static inline void _bt_update_skip_scankeys(IndexScanDesc scan,
+ Relation indexRel);
+static inline bool _bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir);
/*
@@ -1375,6 +1379,376 @@ _bt_next(IndexScanDesc scan, ScanDirection dir)
return true;
}
+/*
+ * _bt_skip() -- Skip items that have the same prefix as the most recently
+ * fetched index tuple.
+ *
+ * The current position is set so that a subsequent call to _bt_next will
+ * fetch the first tuple that differs in the leading 'prefix' keys.
+ *
+ * There are four different kinds of skipping (depending on dir and
+ * indexdir, that are important to distinguish, especially in the presense
+ * of an index condition:
+ *
+ * * Advancing forward and reading forward
+ * simple scan
+ *
+ * * Advancing forward and reading backward
+ * scan inside a cursor fetching backward, when skipping is necessary
+ * right from the start
+ *
+ * * Advancing backward and reading forward
+ * scan with order by desc inside a cursor fetching forward, when
+ * skipping is necessary right from the start
+ *
+ * * Advancing backward and reading backward
+ * simple scan with order by desc
+ *
+ * The current page is searched for the next unique value. If none is found
+ * we will do a scan from the root in order to find the next page with
+ * a unique value.
+ */
+bool
+_bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ BTStack stack;
+ Buffer buf;
+ OffsetNumber offnum;
+ BTScanPosItem *currItem;
+ Relation indexRel = scan->indexRelation;
+
+ /* We want to return tuples, and we need a starting point */
+ Assert(scan->xs_want_itup);
+ Assert(scan->xs_itup);
+
+ /* If skipScanKey is NULL then we initialize it with _bt_mkscankey */
+ if (so->skipScanKey == NULL)
+ {
+ so->skipScanKey = _bt_mkscankey(indexRel, scan->xs_itup);
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->scantid = NULL;
+ }
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /* Check if the next unique key can be found within the current page.
+ * Since we do not lock the current page between jumps, it's possible
+ * that it was splitted since the last time we saw it. This is fine in
+ * case of scanning forward, since page split to the right and we are
+ * still on the left most page. In case of scanning backwards it's
+ * possible to loose some pages and we need to remember the previous
+ * page, and then follow the right link from the current page until we
+ * find the original one.
+ *
+ * Since the whole idea of checking the current page is to protect
+ * ourselves and make more performant statistic mismatch case when
+ * there are too many distinct values for jumping, it's not clear if
+ * the complexity of this solution in case of backward scan is
+ * justified, so for now just avoid it.
+ */
+ if (BufferIsValid(so->currPos.buf) && ScanDirectionIsForward(dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ {
+ bool keyFound = false;
+
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, so->currPos.buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(so->currPos.buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /* Now read the data */
+ keyFound = _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+
+ if (keyFound)
+ {
+ /* set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+ }
+ }
+ else
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ if (BufferIsValid(so->currPos.buf))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ /*
+ * We haven't found scan key within the current page, so let's scan from
+ * the root. Use _bt_search and _bt_binsrch to get the buffer and offset
+ * number
+ */
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /*
+ * Simplest case is when both directions are forward, when we are already
+ * at the next distinct key at the beginning of the series (so everything
+ * else would be done in _bt_readpage)
+ *
+ * The case when both directions are backwards is also simple, but we need
+ * to go one step back, since we need a last element from the previous
+ * series.
+ */
+ if (ScanDirectionIsBackward(dir) && ScanDirectionIsBackward(indexdir))
+ offnum = OffsetNumberPrev(offnum);
+
+ /*
+ * Andvance backward but read forward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can read forward without doing anything else. Otherwise
+ * find previous distinct key and the beginning of it's series and read
+ * forward from there. To do so, go back one step, perform binary search
+ * to find the first item in the series and let _bt_readpage do everything
+ * else.
+ */
+ else if (ScanDirectionIsBackward(dir) && ScanDirectionIsForward(indexdir))
+ {
+ if (!scanstart)
+ {
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* One step back to find a previous value */
+ _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (_bt_next(scan, dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /*
+ * And now find the last item from the sequence for the
+ * current, value with the intention do OffsetNumberNext. As a
+ * result we end up on a first element from the sequence.
+ */
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ else
+ {
+ if (BufferIsValid(so->currPos.buf))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ }
+ }
+ else
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ }
+
+ /*
+ * Advance forward but read backward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can go one step back and read forward without doing
+ * anything else. Otherwise find the next distinct key and the beginning
+ * of it's series, go one step back and read backward from there.
+ *
+ * An interesting situation can happen if one of distinct keys do not pass
+ * a corresponding index condition at all. In this case reading backward
+ * can lead to a previous distinct key being found, creating a loop. To
+ * avoid that check the value to be returned, and jump one more time if
+ * it's the same as at the beginning.
+ */
+ else if (ScanDirectionIsForward(dir) && ScanDirectionIsBackward(indexdir))
+ {
+ if (scanstart)
+ offnum = OffsetNumberPrev(offnum);
+ else
+ {
+ OffsetNumber nextOffset,
+ startOffset;
+
+ IndexTuple startItup = CopyIndexTuple(scan->xs_itup);
+ Page page = BufferGetPage(so->currPos.buf);
+
+ /* We are at the end and need to return */
+ if ((offnum > PageGetMaxOffsetNumber(page)) &
+ (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+
+ nextOffset = startOffset = ItemPointerGetOffsetNumber(&scan->xs_itup->t_tid);
+
+ while (nextOffset == startOffset)
+ {
+ IndexTuple itup;
+
+ /*
+ * Find a next index tuple to update scan key. It could be at
+ * the end, so check for max offset
+ */
+ OffsetNumber maxoff = PageGetMaxOffsetNumber(page);
+ ItemId itemid = PageGetItemId(page, Min(offnum, maxoff));
+
+ CHECK_FOR_INTERRUPTS();
+
+ page = BufferGetPage(so->currPos.buf);
+ scan->xs_itup = (IndexTuple) PageGetItem(page, itemid);
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+
+ _bt_update_skip_scankeys(scan, indexRel);
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ if (BufferIsValid(so->currPos.buf))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ offnum = OffsetNumberPrev(offnum);
+
+ /* Check if _bt_readpage returns already found item */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+
+ currItem = &so->currPos.items[so->currPos.lastItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ nextOffset = ItemPointerGetOffsetNumber(&itup->t_tid);
+
+ /*
+ * To check if we returned the same tuple, try to find a
+ * startItup on the current page. For that we need to update
+ * scankey to match the whole tuple and set nextkey to return
+ * an exact tuple, not the next one. If the nextOffset is the
+ * same as before, it means we are in the loop, return offnum
+ * to the original position and jump further
+ */
+ scan->xs_itup = startItup;
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ so->skipScanKey->keysz = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ so->skipScanKey->nextkey = false;
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey,
+ so->currPos.buf, dir))
+ {
+ startOffset = _bt_binsrch(scan->indexRelation,
+ so->skipScanKey,
+ so->currPos.buf);
+
+ page = BufferGetPage(so->currPos.buf);
+ maxoff = PageGetMaxOffsetNumber(page);
+
+ if (nextOffset == startOffset)
+ offnum = OffsetNumberNext(offnum);
+
+ if ((offnum > maxoff) & (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+
+ /* Return original scankey options */
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ }
+ }
+ }
+
+ /* Now read the data */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to advance to
+ * the next page. Return false if there's no matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ else
+ {
+ /* Drop the lock, and maybe the pin, on the current page */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ /* And set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ return true;
+}
+
/*
* _bt_readpage() -- Load data from current index page into so->currPos
*
@@ -2246,3 +2620,54 @@ _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
so->numKilled = 0; /* just paranoia */
so->markItemIndex = -1; /* ditto */
}
+
+/*
+ * _bt_update_skip_scankeys() -- set up a new values for the existing scankeys
+ * based on the current index tuple
+ */
+static inline void
+_bt_update_skip_scankeys(IndexScanDesc scan, Relation indexRel)
+{
+ TupleDesc itupdesc;
+ int indnkeyatts,
+ i;
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ ScanKey scankeys = so->skipScanKey->scankeys;
+
+ itupdesc = RelationGetDescr(indexRel);
+ indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ for (i = 0; i < indnkeyatts; i++)
+ {
+ Datum datum;
+ bool null;
+ int flags;
+
+ datum = index_getattr(scan->xs_itup, i + 1, itupdesc, &null);
+ flags = (null ? SK_ISNULL : 0) |
+ (indexRel->rd_indoption[i] << SK_BT_INDOPTION_SHIFT);
+ scankeys[i].sk_flags = flags;
+ scankeys[i].sk_argument = datum;
+ }
+}
+
+/*
+ * _bt_scankey_within_page() -- check if the provided scankey could be found
+ * within a page, specified by the buffer.
+ */
+static inline bool
+_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir)
+{
+ OffsetNumber low, high;
+ Page page = BufferGetPage(buf);
+ BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ low = P_FIRSTDATAKEY(opaque);
+ high = PageGetMaxOffsetNumber(page);
+
+ if (unlikely(high < low))
+ return false;
+
+ return (_bt_compare(scan->indexRelation, key, page, low) > 0 &&
+ _bt_compare(scan->indexRelation, key, page, high) < 1);
+}
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4924ae1c59..fa09a4685e 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -68,6 +68,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d573..60cd801247 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -130,6 +130,7 @@ static void ExplainDummyGroup(const char *objtype, const char *labelname,
static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1058,6 +1059,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1380,6 +1397,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1390,6 +1409,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1599,6 +1620,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1612,6 +1635,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..76330f7906 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -65,6 +65,13 @@ IndexOnlyNext(IndexOnlyScanState *node)
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +79,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,14 +122,50 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, direction)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* We can skip the heap fetch if the TID references a heap page on
@@ -250,6 +293,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +549,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 54ad62bb7f..e0cfd710c4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 16083e7a7e..5f723cda4b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 551ce6c41c..028d03a56d 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1820,6 +1820,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1839,6 +1840,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index b5a0033721..710edf160a 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -124,6 +124,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index dff826a828..7b32f2cc7e 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -175,12 +175,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2910,7 +2912,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2921,7 +2924,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5184,7 +5188,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5201,6 +5206,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5213,7 +5219,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5228,6 +5235,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 984fca0696..c84388e6f7 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4834,6 +4834,82 @@ create_distinct_paths(PlannerInfo *root,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+
+ /* Consider index skip scan as well */
+ if (enable_indexskipscan &&
+ IsA(path, IndexPath) &&
+ ((IndexPath *) path)->indexinfo->amcanskip &&
+ root->distinct_pathkeys != NIL)
+ {
+ ListCell *lc;
+ IndexOptInfo *index = NULL;
+ bool different_columns_order = false,
+ not_empty_qual = false;
+ int i = 0;
+ int distinctPrefixKeys;
+
+ Assert(path->pathtype == T_IndexOnlyScan ||
+ path->pathtype == T_IndexScan);
+
+ index = ((IndexPath *) path)->indexinfo;
+ distinctPrefixKeys = list_length(root->query_uniquekeys);
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniquekey = (UniqueKey *) lfirst(lc);
+ EquivalenceMember *em =
+ lfirst_node(EquivalenceMember,
+ list_head(uniquekey->eq_clause->ec_members));
+ Var *var = (Var *) em->em_expr;
+
+ Assert(i < index->ncolumns);
+
+ for (i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ /*
+ * XXX: In case of index scan quals evaluation happens
+ * after ExecScanFetch, which means skip results could be
+ * fitered out. Consider the following query:
+ *
+ * select distinct (a, b) a, b, c from t where c < 100;
+ *
+ * Skip scan returns one tuple for one distinct set of (a,
+ * b) with arbitrary one of c, so if the choosed c does
+ * not match the qual and there is any c that matches the
+ * qual, we miss that tuple.
+ */
+ if (path->pathtype == T_IndexScan &&
+ parse->jointree != NULL &&
+ parse->jointree->quals != NULL &&
+ list_length((List *) parse->jointree->quals) != 0)
+ not_empty_qual = true;
+
+ if (!different_columns_order && !not_empty_qual)
+ {
+ add_path(distinct_rel, (Path *)
+ create_skipscan_unique_path(root,
+ distinct_rel,
+ path,
+ distinctPrefixKeys,
+ numDistinctRows));
+ }
+ }
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index a006dbbe9c..2fb18fb372 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2915,6 +2915,46 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *basepath,
+ int distinctPrefixKeys,
+ double numGroups)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /* The size of the prefix we'll use for skipping. */
+ Assert(pathnode->indexinfo->amcanskip);
+ Assert(distinctPrefixKeys > 0);
+ /*Assert(distinctPrefixKeys <= list_length(pathnode->path.pathkeys));*/
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ /*
+ * The cost to skip to each distinct value should be roughly the same as
+ * the cost of finding the first key times the number of distinct values
+ * we expect to find.
+ */
+ pathnode->path.startup_cost = basepath->startup_cost;
+ pathnode->path.total_cost = basepath->startup_cost * numGroups;
+ pathnode->path.rows = numGroups;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index d82fc5ab8b..f65b299f37 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -271,6 +271,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index e5f8a1301f..3258b1d007 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -918,6 +918,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e1048c0047..a002ee2143 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -353,6 +353,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 3b3e22f73d..3d39cd9d07 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -229,6 +236,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 7e9364a50c..815de4e4dd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 20ace69dab..e098c6a1ab 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -662,6 +662,9 @@ typedef struct BTScanOpaqueData
*/
int markItemIndex; /* itemIndex, or -1 if not valid */
+ /* Work space for _bt_skip */
+ BTScanInsert skipScanKey; /* used to control skipping */
+
/* keep these last in struct for efficiency */
BTScanPosData currPos; /* current position data */
BTScanPosData markPos; /* marked position, if any */
@@ -793,6 +796,8 @@ extern OffsetNumber _bt_binsrch_insert(Relation rel, BTInsertState insertstate);
extern int32 _bt_compare(Relation rel, BTScanInsert key, Page page, OffsetNumber offnum);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
+extern bool _bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
Snapshot snapshot);
@@ -817,6 +822,8 @@ extern void _bt_end_vacuum_callback(int code, Datum arg);
extern Size BTreeShmemSize(void);
extern void BTreeShmemInit(void);
extern bytea *btoptions(Datum reloptions, bool validate);
+extern bool btskip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index eaea1f3b0c..94cf92d07f 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1423,6 +1423,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1452,6 +1454,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1470,6 +1474,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 4e329f0fb5..b0ff9ca3a8 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -839,6 +839,7 @@ struct IndexOptInfo
bool amsearchnulls; /* can AM search for NULL/NOT NULL entries? */
bool amhasgettuple; /* does AM have amgettuple interface? */
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
+ bool amcanskip; /* can AM skip duplicate values? */
bool amcanparallel; /* does AM support parallel scan? */
/* Rather than include amapi.h here, we declare amcostestimate like this */
void (*amcostestimate) (); /* AM's cost estimator */
@@ -1189,6 +1190,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1201,6 +1205,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 32c0d87f80..03a00e8e1d 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index cb012ba198..847f34f02b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index f75ff6f323..6c8c9dadbb 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -201,6 +201,11 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *subpath,
+ int numCols,
+ double numGroups);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/test/regress/expected/select_distinct.out b/src/test/regress/expected/select_distinct.out
index f3696c6d1d..51e12ac925 100644
--- a/src/test/regress/expected/select_distinct.out
+++ b/src/test/regress/expected/select_distinct.out
@@ -244,3 +244,508 @@ SELECT null IS NOT DISTINCT FROM null as "yes";
t
(1 row)
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+SELECT DISTINCT a FROM distinct_a;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+ a
+---
+ 1
+(1 row)
+
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+ QUERY PLAN
+--------------------------------------------------------
+ Index Only Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+DROP INDEX distinct_a_b_a;
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+FETCH FROM c;
+ a | b
+---+---
+ 1 | 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+END;
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+FETCH FROM c;
+ a | b
+---+-------
+ 5 | 10000
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+END;
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+ QUERY PLAN
+--------------------------------------------------------------
+ Index Only Scan using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 1 | 2
+ 3 | 1 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 1 | 2
+ 1 | 1 | 2
+(2 rows)
+
+END;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+ QUERY PLAN
+-----------------------------------------------------------------------
+ Index Only Scan Backward using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 2 | 2
+ 1 | 2 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 2 | 2
+ 3 | 2 | 2
+(2 rows)
+
+END;
+DROP TABLE distinct_abc;
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+ 2 | 1 | 10
+ 3 | 1 | 10
+ 4 | 1 | 10
+ 5 | 1 | 10
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ QUERY PLAN
+---------------------------------------------------
+ Index Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ QUERY PLAN
+-----------------------------------------------------
+ Unique
+ -> Bitmap Heap Scan on distinct_a
+ Recheck Cond: (a = 1)
+ -> Bitmap Index Scan on distinct_a_a_b_idx
+ Index Cond: (a = 1)
+(5 rows)
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ QUERY PLAN
+---------------------------------------------------------
+ Unique
+ -> Index Scan using distinct_a_a_b_idx on distinct_a
+ Index Cond: (b = 2)
+ Filter: (c = 10)
+(4 rows)
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+ a | a
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 3
+ 4 | 4
+ 5 | 5
+(5 rows)
+
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+ a | ?column?
+---+----------
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+FETCH FROM c;
+ a
+---
+ 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a
+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+END;
+DROP TABLE distinct_a;
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 9999
+ 1 | 10000
+(5 rows)
+
+DROP TABLE distinct_visibility;
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Index Only Scan using distinct_boundaries_a_b_c_idx on distinct_boundaries
+ Skip scan: true
+ Index Cond: ((b >= 1) AND (c = 0))
+(3 rows)
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ a | b | c
+---+---+---
+ 1 | 2 | 0
+ 2 | 2 | 0
+ 3 | 2 | 0
+ 4 | 2 | 0
+ 5 | 2 | 0
+(5 rows)
+
+DROP TABLE distinct_boundaries;
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a1c90eb905..bd3b373515 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -78,6 +78,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_hashjoin | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -89,7 +90,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(17 rows)
+(18 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/sql/select_distinct.sql b/src/test/regress/sql/select_distinct.sql
index a605e86449..4c8a50d153 100644
--- a/src/test/regress/sql/select_distinct.sql
+++ b/src/test/regress/sql/select_distinct.sql
@@ -73,3 +73,189 @@ SELECT 1 IS NOT DISTINCT FROM 2 as "no";
SELECT 2 IS NOT DISTINCT FROM 2 as "yes";
SELECT 2 IS NOT DISTINCT FROM null as "no";
SELECT null IS NOT DISTINCT FROM null as "yes";
+
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+
+SELECT DISTINCT a FROM distinct_a;
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+DROP INDEX distinct_a_b_a;
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+DROP TABLE distinct_abc;
+
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+DROP TABLE distinct_a;
+
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DROP TABLE distinct_visibility;
+
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+DROP TABLE distinct_boundaries;
--
2.21.0
--cxnum6r77tfmznfe--
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v33 2/2] Index skip scan
@ 2019-11-15 14:46 jesperpedersen <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: jesperpedersen @ 2019-11-15 14:46 UTC (permalink / raw)
Implementation of Index Skip Scan (see Loose Index Scan in the wiki [1])
on top of IndexOnlyScan and IndexScan. To make it suitable for both
situations when there are small number of distinct values and
significant amount of distinct values the following approach is taken -
instead of searching from the root for every value we're searching for
then first on the current page, and then if not found continue searching
from the root.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 +
doc/src/sgml/indexam.sgml | 63 ++
doc/src/sgml/indices.sgml | 23 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 +
src/backend/access/nbtree/nbtree.c | 13 +
src/backend/access/nbtree/nbtsearch.c | 458 +++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +
src/backend/commands/portalcmds.c | 2 +
src/backend/executor/nodeIndexonlyscan.c | 51 +-
src/backend/executor/nodeIndexscan.c | 56 +-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/planner.c | 77 +++
src/backend/optimizer/util/pathnode.c | 40 ++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/nbtree.h | 7 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/parsenodes.h | 1 +
src/include/nodes/pathnodes.h | 5 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 5 +
src/test/regress/expected/select_distinct.out | 601 ++++++++++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/select_distinct.sql | 248 ++++++++
39 files changed, 1772 insertions(+), 11 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 0104d02f67..a018b7f3d0 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -133,6 +133,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e07dc01e80..36ba75b077 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4517,6 +4517,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 37f8d8760a..a726d80878 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -148,6 +148,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -691,6 +692,68 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ ScanDirection indexdir,
+ bool scanstart,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>indexdir</parameter></term>
+ <listitem>
+ <para>
+ The index direction, in which data must be read.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>scanstart</parameter></term>
+ <listitem>
+ <para>
+ Whether or not it is a start of the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index c54bf0dbbd..c429d98fc7 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1254,6 +1254,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 2e8f67ef10..4db31bb211 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -113,6 +113,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a7e55caf28..8dd1d30d2a 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -65,6 +65,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index aefc302ed2..8c692f7fb4 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 4871b7ff4d..e5fa4c7864 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 01539b6bd6..1047a35ade 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -730,6 +731,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 5254bc7ef5..8fde56fe60 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -132,6 +132,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
+ amroutine->amskip = btskip;
amroutine->amcostestimate = btcostestimate;
amroutine->amoptions = btoptions;
amroutine->amproperty = btproperty;
@@ -381,6 +382,8 @@ btbeginscan(Relation rel, int nkeys, int norderbys)
*/
so->currTuples = so->markTuples = NULL;
+ so->skipScanKey = NULL;
+
scan->xs_itupdesc = RelationGetDescr(rel);
scan->opaque = so;
@@ -448,6 +451,16 @@ btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
_bt_preprocess_array_keys(scan);
}
+/*
+ * btskip() -- skip to the beginning of the next key prefix
+ */
+bool
+btskip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix)
+{
+ return _bt_skip(scan, direction, indexdir, start, prefix);
+}
+
/*
* btendscan() -- close down a scan
*/
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index c573814f01..8b406416fd 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -37,6 +37,10 @@ static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
static Buffer _bt_walk_left(Relation rel, Buffer buf, Snapshot snapshot);
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
+static inline void _bt_update_skip_scankeys(IndexScanDesc scan,
+ Relation indexRel);
+static inline bool _bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir);
/*
@@ -1375,6 +1379,409 @@ _bt_next(IndexScanDesc scan, ScanDirection dir)
return true;
}
+/*
+ * _bt_skip() -- Skip items that have the same prefix as the most recently
+ * fetched index tuple.
+ *
+ * The current position is set so that a subsequent call to _bt_next will
+ * fetch the first tuple that differs in the leading 'prefix' keys.
+ *
+ * There are four different kinds of skipping (depending on dir and
+ * indexdir, that are important to distinguish, especially in the presense
+ * of an index condition:
+ *
+ * * Advancing forward and reading forward
+ * simple scan
+ *
+ * * Advancing forward and reading backward
+ * scan inside a cursor fetching backward, when skipping is necessary
+ * right from the start
+ *
+ * * Advancing backward and reading forward
+ * scan with order by desc inside a cursor fetching forward, when
+ * skipping is necessary right from the start
+ *
+ * * Advancing backward and reading backward
+ * simple scan with order by desc
+ *
+ * The current page is searched for the next unique value. If none is found
+ * we will do a scan from the root in order to find the next page with
+ * a unique value.
+ */
+bool
+_bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ BTStack stack;
+ Buffer buf;
+ OffsetNumber offnum;
+ BTScanPosItem *currItem;
+ Relation indexRel = scan->indexRelation;
+
+ /* We want to return tuples, and we need a starting point */
+ Assert(scan->xs_want_itup);
+ Assert(scan->xs_itup);
+
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ /* If skipScanKey is NULL then we initialize it with _bt_mkscankey */
+ if (so->skipScanKey == NULL)
+ {
+ so->skipScanKey = _bt_mkscankey(indexRel, scan->xs_itup);
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->scantid = NULL;
+ }
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /* Check if the next unique key can be found within the current page.
+ * Since we do not lock the current page between jumps, it's possible
+ * that it was splitted since the last time we saw it. This is fine in
+ * case of scanning forward, since page split to the right and we are
+ * still on the left most page. In case of scanning backwards it's
+ * possible to loose some pages and we need to remember the previous
+ * page, and then follow the right link from the current page until we
+ * find the original one.
+ *
+ * Since the whole idea of checking the current page is to protect
+ * ourselves and make more performant statistic mismatch case when
+ * there are too many distinct values for jumping, it's not clear if
+ * the complexity of this solution in case of backward scan is
+ * justified, so for now just avoid it.
+ */
+ if (BufferIsValid(so->currPos.buf) && ScanDirectionIsForward(dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ {
+ bool keyFound = false;
+
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, so->currPos.buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(so->currPos.buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /* Now read the data */
+ keyFound = _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+
+ if (keyFound)
+ {
+ /* set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+ }
+ }
+ else
+ {
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+ }
+
+ if (BufferIsValid(so->currPos.buf))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ /*
+ * We haven't found scan key within the current page, so let's scan from
+ * the root. Use _bt_search and _bt_binsrch to get the buffer and offset
+ * number
+ */
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /*
+ * Simplest case is when both directions are forward, when we are already
+ * at the next distinct key at the beginning of the series (so everything
+ * else would be done in _bt_readpage)
+ *
+ * The case when both directions are backwards is also simple, but we need
+ * to go one step back, since we need a last element from the previous
+ * series.
+ */
+ if (ScanDirectionIsBackward(dir) && ScanDirectionIsBackward(indexdir))
+ offnum = OffsetNumberPrev(offnum);
+
+ /*
+ * Andvance backward but read forward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can read forward without doing anything else. Otherwise
+ * find previous distinct key and the beginning of it's series and read
+ * forward from there. To do so, go back one step, perform binary search
+ * to find the first item in the series and let _bt_readpage do everything
+ * else.
+ */
+ else if (ScanDirectionIsBackward(dir) && ScanDirectionIsForward(indexdir))
+ {
+ if (!scanstart)
+ {
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* One step back to find a previous value */
+ _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (_bt_next(scan, dir))
+ {
+ LockBuffer(so->currPos.buf, BT_READ);
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /*
+ * And now find the last item from the sequence for the
+ * current, value with the intention do OffsetNumberNext. As a
+ * result we end up on a first element from the sequence.
+ */
+ if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ else
+ {
+ if (BufferIsValid(so->currPos.buf))
+ {
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ }
+ }
+ else
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ }
+
+ /*
+ * Advance forward but read backward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can go one step back and read forward without doing
+ * anything else. Otherwise find the next distinct key and the beginning
+ * of it's series, go one step back and read backward from there.
+ *
+ * An interesting situation can happen if one of distinct keys do not pass
+ * a corresponding index condition at all. In this case reading backward
+ * can lead to a previous distinct key being found, creating a loop. To
+ * avoid that check the value to be returned, and jump one more time if
+ * it's the same as at the beginning.
+ */
+ else if (ScanDirectionIsForward(dir) && ScanDirectionIsBackward(indexdir))
+ {
+ if (scanstart)
+ offnum = OffsetNumberPrev(offnum);
+ else
+ {
+ OffsetNumber nextOffset,
+ startOffset,
+ jumpOffset;
+
+ IndexTuple startItup = CopyIndexTuple(scan->xs_itup);
+ Page page = BufferGetPage(so->currPos.buf);
+
+ /* We are at the end and need to return */
+ if ((offnum > PageGetMaxOffsetNumber(page)) &
+ (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+
+ nextOffset = startOffset = ItemPointerGetOffsetNumber(&scan->xs_itup->t_tid);
+
+ while (nextOffset == startOffset)
+ {
+ IndexTuple itup;
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * Find a next index tuple to update scan key. It could be at
+ * the end, so check for max offset
+ */
+ if (!_bt_readpage(scan, ForwardScanDirection, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ LockBuffer(so->currPos.buf, BT_READ);
+ }
+
+ currItem = &so->currPos.items[so->currPos.firstItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ scan->xs_itup = itup;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+
+ _bt_update_skip_scankeys(scan, indexRel);
+ if (BufferIsValid(so->currPos.buf))
+ {
+ if (so->numKilled > 0)
+ _bt_killitems(scan);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ jumpOffset = offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ offnum = OffsetNumberPrev(offnum);
+
+ so->currPos.moreLeft = true;
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, indexdir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ LockBuffer(so->currPos.buf, BT_READ);
+ }
+
+ currItem = &so->currPos.items[so->currPos.lastItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ nextOffset = ItemPointerGetOffsetNumber(&itup->t_tid);
+
+ /*
+ * To check if we returned the same tuple, try to find a
+ * startItup on the current page. For that we need to update
+ * scankey to match the whole tuple and set nextkey to return
+ * an exact tuple, not the next one. If the nextOffset is the
+ * same as before, it means we are in the loop, return offnum
+ * to the original position and jump further
+ */
+ scan->xs_itup = startItup;
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ so->skipScanKey->keysz = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ so->skipScanKey->nextkey = false;
+
+ if (_bt_scankey_within_page(scan, so->skipScanKey,
+ so->currPos.buf, dir))
+ {
+ OffsetNumber maxoff;
+ startOffset = _bt_binsrch(scan->indexRelation,
+ so->skipScanKey,
+ so->currPos.buf);
+
+ page = BufferGetPage(so->currPos.buf);
+ maxoff = PageGetMaxOffsetNumber(page);
+
+ if (nextOffset <= startOffset)
+ {
+ offnum = jumpOffset;
+ nextOffset = startOffset;
+ }
+
+ if ((offnum > maxoff) & (so->currPos.nextPage == P_NONE))
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+
+ /* Return original scankey options */
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ }
+ }
+ }
+
+ /* Now read the data */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to advance to
+ * the next page. Return false if there's no matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ else
+ {
+ /* Drop the lock, and maybe the pin, on the current page */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ /* And set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+}
+
/*
* _bt_readpage() -- Load data from current index page into so->currPos
*
@@ -2246,3 +2653,54 @@ _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
so->numKilled = 0; /* just paranoia */
so->markItemIndex = -1; /* ditto */
}
+
+/*
+ * _bt_update_skip_scankeys() -- set up a new values for the existing scankeys
+ * based on the current index tuple
+ */
+static inline void
+_bt_update_skip_scankeys(IndexScanDesc scan, Relation indexRel)
+{
+ TupleDesc itupdesc;
+ int indnkeyatts,
+ i;
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ ScanKey scankeys = so->skipScanKey->scankeys;
+
+ itupdesc = RelationGetDescr(indexRel);
+ indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ for (i = 0; i < indnkeyatts; i++)
+ {
+ Datum datum;
+ bool null;
+ int flags;
+
+ datum = index_getattr(scan->xs_itup, i + 1, itupdesc, &null);
+ flags = (null ? SK_ISNULL : 0) |
+ (indexRel->rd_indoption[i] << SK_BT_INDOPTION_SHIFT);
+ scankeys[i].sk_flags = flags;
+ scankeys[i].sk_argument = datum;
+ }
+}
+
+/*
+ * _bt_scankey_within_page() -- check if the provided scankey could be found
+ * within a page, specified by the buffer.
+ */
+static inline bool
+_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir)
+{
+ OffsetNumber low, high;
+ Page page = BufferGetPage(buf);
+ BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ low = P_FIRSTDATAKEY(opaque);
+ high = PageGetMaxOffsetNumber(page);
+
+ if (unlikely(high < low))
+ return false;
+
+ return (_bt_compare(scan->indexRelation, key, page, low) > 0 &&
+ _bt_compare(scan->indexRelation, key, page, high) < 1);
+}
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4924ae1c59..fa09a4685e 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -68,6 +68,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index c367c750b1..a7dd874531 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -141,6 +141,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1052,6 +1053,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1386,6 +1403,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1396,6 +1415,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1655,6 +1676,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1668,6 +1691,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/commands/portalcmds.c b/src/backend/commands/portalcmds.c
index 7e5c805a1e..b9999ca3a9 100644
--- a/src/backend/commands/portalcmds.c
+++ b/src/backend/commands/portalcmds.c
@@ -89,6 +89,8 @@ PerformCursorOpen(ParseState *pstate, DeclareCursorStmt *cstmt, ParamListInfo pa
if (query->commandType != CMD_SELECT)
elog(ERROR, "non-SELECT statement in DECLARE CURSOR");
+ query->hasDeclareCursor = true;
+
/* Plan the query, applying the specified options */
plan = pg_plan_query(query, cstmt->options, params);
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..76330f7906 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -65,6 +65,13 @@ IndexOnlyNext(IndexOnlyScanState *node)
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +79,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,14 +122,50 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, direction)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* We can skip the heap fetch if the TID references a heap page on
@@ -250,6 +293,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +549,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 54ad62bb7f..e0cfd710c4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 16083e7a7e..5f723cda4b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 551ce6c41c..028d03a56d 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1820,6 +1820,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1839,6 +1840,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index b5a0033721..710edf160a 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -124,6 +124,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index dff826a828..7b32f2cc7e 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -175,12 +175,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2910,7 +2912,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2921,7 +2924,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5184,7 +5188,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5201,6 +5206,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5213,7 +5219,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5228,6 +5235,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 984fca0696..777ae3ee0f 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4834,6 +4834,83 @@ create_distinct_paths(PlannerInfo *root,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+
+ /* Consider index skip scan as well */
+ if (enable_indexskipscan &&
+ IsA(path, IndexPath) &&
+ root->parse->hasDeclareCursor == false &&
+ ((IndexPath *) path)->indexinfo->amcanskip &&
+ root->distinct_pathkeys != NIL)
+ {
+ ListCell *lc;
+ IndexOptInfo *index = NULL;
+ bool different_columns_order = false,
+ not_empty_qual = false;
+ int i = 0;
+ int distinctPrefixKeys;
+
+ Assert(path->pathtype == T_IndexOnlyScan ||
+ path->pathtype == T_IndexScan);
+
+ index = ((IndexPath *) path)->indexinfo;
+ distinctPrefixKeys = list_length(root->query_uniquekeys);
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniquekey = (UniqueKey *) lfirst(lc);
+ EquivalenceMember *em =
+ lfirst_node(EquivalenceMember,
+ list_head(uniquekey->eq_clause->ec_members));
+ Var *var = (Var *) em->em_expr;
+
+ Assert(i < index->ncolumns);
+
+ for (i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ /*
+ * XXX: In case of index scan quals evaluation happens
+ * after ExecScanFetch, which means skip results could be
+ * fitered out. Consider the following query:
+ *
+ * select distinct (a, b) a, b, c from t where c < 100;
+ *
+ * Skip scan returns one tuple for one distinct set of (a,
+ * b) with arbitrary one of c, so if the choosed c does
+ * not match the qual and there is any c that matches the
+ * qual, we miss that tuple.
+ */
+ if (path->pathtype == T_IndexScan &&
+ parse->jointree != NULL &&
+ parse->jointree->quals != NULL &&
+ list_length((List *) parse->jointree->quals) != 0)
+ not_empty_qual = true;
+
+ if (!different_columns_order && !not_empty_qual)
+ {
+ add_path(distinct_rel, (Path *)
+ create_skipscan_unique_path(root,
+ distinct_rel,
+ path,
+ distinctPrefixKeys,
+ numDistinctRows));
+ }
+ }
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index a006dbbe9c..2fb18fb372 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2915,6 +2915,46 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *basepath,
+ int distinctPrefixKeys,
+ double numGroups)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /* The size of the prefix we'll use for skipping. */
+ Assert(pathnode->indexinfo->amcanskip);
+ Assert(distinctPrefixKeys > 0);
+ /*Assert(distinctPrefixKeys <= list_length(pathnode->path.pathkeys));*/
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ /*
+ * The cost to skip to each distinct value should be roughly the same as
+ * the cost of finding the first key times the number of distinct values
+ * we expect to find.
+ */
+ pathnode->path.startup_cost = basepath->startup_cost;
+ pathnode->path.total_cost = basepath->startup_cost * numGroups;
+ pathnode->path.rows = numGroups;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index d82fc5ab8b..f65b299f37 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -271,6 +271,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index cacbe904db..7c71ee4499 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -923,6 +923,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e1048c0047..a002ee2143 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -353,6 +353,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 3b3e22f73d..3d39cd9d07 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -229,6 +236,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 7e9364a50c..815de4e4dd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 20ace69dab..e098c6a1ab 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -662,6 +662,9 @@ typedef struct BTScanOpaqueData
*/
int markItemIndex; /* itemIndex, or -1 if not valid */
+ /* Work space for _bt_skip */
+ BTScanInsert skipScanKey; /* used to control skipping */
+
/* keep these last in struct for efficiency */
BTScanPosData currPos; /* current position data */
BTScanPosData markPos; /* marked position, if any */
@@ -793,6 +796,8 @@ extern OffsetNumber _bt_binsrch_insert(Relation rel, BTInsertState insertstate);
extern int32 _bt_compare(Relation rel, BTScanInsert key, Page page, OffsetNumber offnum);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
+extern bool _bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
Snapshot snapshot);
@@ -817,6 +822,8 @@ extern void _bt_end_vacuum_callback(int code, Datum arg);
extern Size BTreeShmemSize(void);
extern void BTreeShmemInit(void);
extern bytea *btoptions(Datum reloptions, bool validate);
+extern bool btskip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 1f6f5bbc20..2c6acc160a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1423,6 +1423,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1452,6 +1454,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1470,6 +1474,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index da0706add5..f69468d56a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -131,6 +131,7 @@ typedef struct Query
bool hasModifyingCTE; /* has INSERT/UPDATE/DELETE in WITH */
bool hasForUpdate; /* FOR [KEY] UPDATE/SHARE was specified */
bool hasRowSecurity; /* rewriter has applied some RLS policy */
+ bool hasDeclareCursor; /* has declaring cursor */
List *cteList; /* WITH list (of CommonTableExpr's) */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 4e329f0fb5..b0ff9ca3a8 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -839,6 +839,7 @@ struct IndexOptInfo
bool amsearchnulls; /* can AM search for NULL/NOT NULL entries? */
bool amhasgettuple; /* does AM have amgettuple interface? */
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
+ bool amcanskip; /* can AM skip duplicate values? */
bool amcanparallel; /* does AM support parallel scan? */
/* Rather than include amapi.h here, we declare amcostestimate like this */
void (*amcostestimate) (); /* AM's cost estimator */
@@ -1189,6 +1190,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1201,6 +1205,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 32c0d87f80..03a00e8e1d 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index cb012ba198..847f34f02b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index f75ff6f323..6c8c9dadbb 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -201,6 +201,11 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *subpath,
+ int numCols,
+ double numGroups);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/test/regress/expected/select_distinct.out b/src/test/regress/expected/select_distinct.out
index f3696c6d1d..259db10c81 100644
--- a/src/test/regress/expected/select_distinct.out
+++ b/src/test/regress/expected/select_distinct.out
@@ -244,3 +244,604 @@ SELECT null IS NOT DISTINCT FROM null as "yes";
t
(1 row)
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+SELECT DISTINCT a FROM distinct_a;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+ a
+---
+ 1
+(1 row)
+
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+ QUERY PLAN
+--------------------------------------------------------
+ Index Only Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+DROP INDEX distinct_a_b_a;
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+FETCH FROM c;
+ a | b
+---+---
+ 1 | 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+END;
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+FETCH FROM c;
+ a | b
+---+-------
+ 5 | 10000
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+END;
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+ QUERY PLAN
+--------------------------------------------------------------
+ Index Only Scan using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 1 | 2
+ 3 | 1 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 1 | 2
+ 1 | 1 | 2
+(2 rows)
+
+END;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+ QUERY PLAN
+-----------------------------------------------------------------------
+ Index Only Scan Backward using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 2 | 2
+ 1 | 2 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 2 | 2
+ 3 | 2 | 2
+(2 rows)
+
+END;
+DROP TABLE distinct_abc;
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+ 2 | 1 | 10
+ 3 | 1 | 10
+ 4 | 1 | 10
+ 5 | 1 | 10
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ QUERY PLAN
+---------------------------------------------------
+ Index Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ QUERY PLAN
+-----------------------------------------------------
+ Unique
+ -> Bitmap Heap Scan on distinct_a
+ Recheck Cond: (a = 1)
+ -> Bitmap Index Scan on distinct_a_a_b_idx
+ Index Cond: (a = 1)
+(5 rows)
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ QUERY PLAN
+---------------------------------------------------------
+ Unique
+ -> Index Scan using distinct_a_a_b_idx on distinct_a
+ Index Cond: (b = 2)
+ Filter: (c = 10)
+(4 rows)
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+ a | a
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 3
+ 4 | 4
+ 5 | 5
+(5 rows)
+
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+ a | ?column?
+---+----------
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+FETCH FROM c;
+ a
+---
+ 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a
+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+END;
+DROP TABLE distinct_a;
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 9999
+ 1 | 10000
+(5 rows)
+
+DROP TABLE distinct_visibility;
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Index Only Scan using distinct_boundaries_a_b_c_idx on distinct_boundaries
+ Skip scan: true
+ Index Cond: ((b >= 1) AND (c = 0))
+(3 rows)
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ a | b | c
+---+---+---
+ 1 | 2 | 0
+ 2 | 2 | 0
+ 3 | 2 | 0
+ 4 | 2 | 0
+ 5 | 2 | 0
+(5 rows)
+
+DROP TABLE distinct_boundaries;
+-- test tuple killing
+-- DESC ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed where a = 3;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 5 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 1 | 1000 | 0 | 10
+(4 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 1 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 5 | 1000 | 0 | 10
+(4 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
+-- regular ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed where a = 3;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a, b;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+---+---+----
+ 1 | 1 | 1 | 10
+ 2 | 1 | 1 | 10
+ 4 | 1 | 1 | 10
+ 5 | 1 | 1 | 10
+(4 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+---+---+----
+ 5 | 1 | 1 | 10
+ 4 | 1 | 1 | 10
+ 2 | 1 | 1 | 10
+ 1 | 1 | 1 | 10
+(4 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
+-- partial delete
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+CREATE INDEX ON distinct_killed (a, b, c, d);
+DELETE FROM distinct_killed WHERE a = 3 AND b <= 999;
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 5 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 3 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 1 | 1000 | 0 | 10
+(5 rows)
+
+ FETCH BACKWARD ALL FROM c;
+ a | b | c | d
+---+------+---+----
+ 1 | 1000 | 0 | 10
+ 2 | 1000 | 0 | 10
+ 3 | 1000 | 0 | 10
+ 4 | 1000 | 0 | 10
+ 5 | 1000 | 0 | 10
+(5 rows)
+
+COMMIT;
+DROP TABLE distinct_killed;
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a1c90eb905..bd3b373515 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -78,6 +78,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_hashjoin | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -89,7 +90,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(17 rows)
+(18 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/sql/select_distinct.sql b/src/test/regress/sql/select_distinct.sql
index a605e86449..843efeb28f 100644
--- a/src/test/regress/sql/select_distinct.sql
+++ b/src/test/regress/sql/select_distinct.sql
@@ -73,3 +73,251 @@ SELECT 1 IS NOT DISTINCT FROM 2 as "no";
SELECT 2 IS NOT DISTINCT FROM 2 as "yes";
SELECT 2 IS NOT DISTINCT FROM null as "no";
SELECT null IS NOT DISTINCT FROM null as "yes";
+
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+
+SELECT DISTINCT a FROM distinct_a;
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+DROP INDEX distinct_a_b_a;
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+DROP TABLE distinct_abc;
+
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+DROP TABLE distinct_a;
+
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DROP TABLE distinct_visibility;
+
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+DROP TABLE distinct_boundaries;
+
+-- test tuple killing
+
+-- DESC ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed where a = 3;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
+
+-- regular ordering
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed where a = 3;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a, b;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
+
+-- partial delete
+CREATE TABLE distinct_killed AS
+ SELECT a, b, b % 2 AS c, 10 AS d
+ FROM generate_series(1, 5) a,
+ generate_series(1,1000) b;
+
+CREATE INDEX ON distinct_killed (a, b, c, d);
+
+DELETE FROM distinct_killed WHERE a = 3 AND b <= 999;
+
+BEGIN;
+ DECLARE c SCROLL CURSOR FOR
+ SELECT DISTINCT ON (a) a,b,c,d
+ FROM distinct_killed ORDER BY a DESC, b DESC;
+ FETCH FORWARD ALL FROM c;
+ FETCH BACKWARD ALL FROM c;
+COMMIT;
+
+DROP TABLE distinct_killed;
--
2.21.0
--ezqb2fwnenb5aigz--
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH 2/2] Index skip scan
@ 2020-01-20 13:01 jesperpedersen <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: jesperpedersen @ 2020-01-20 13:01 UTC (permalink / raw)
Implementation of Index Skip Scan (see Loose Index Scan in the wiki [1])
on top of IndexOnlyScan and IndexScan. To make it suitable for both
situations when there are small number of distinct values and
significant amount of distinct values the following approach is taken -
instead of searching from the root for every value we're searching for
then first on the current page, and then if not found continue searching
from the root.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 +
doc/src/sgml/indexam.sgml | 63 +++
doc/src/sgml/indices.sgml | 23 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 +
src/backend/access/nbtree/nbtree.c | 13 +
src/backend/access/nbtree/nbtsearch.c | 366 +++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +
src/backend/executor/nodeIndexonlyscan.c | 51 +-
src/backend/executor/nodeIndexscan.c | 56 +-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/planner.c | 76 +++
src/backend/optimizer/util/pathnode.c | 40 ++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/nbtree.h | 7 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 5 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 5 +
src/test/regress/expected/select_distinct.out | 505 ++++++++++++++++++
src/test/regress/expected/sysviews.out | 3 +-
src/test/regress/sql/select_distinct.sql | 186 +++++++
37 files changed, 1518 insertions(+), 11 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 0104d02f67..a018b7f3d0 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -133,6 +133,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 3ccacd528b..f99e702364 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4517,6 +4517,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 37f8d8760a..a726d80878 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -148,6 +148,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -691,6 +692,68 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ ScanDirection indexdir,
+ bool scanstart,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>indexdir</parameter></term>
+ <listitem>
+ <para>
+ The index direction, in which data must be read.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>scanstart</parameter></term>
+ <listitem>
+ <para>
+ Whether or not it is a start of the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index c54bf0dbbd..c429d98fc7 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1254,6 +1254,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 2e8f67ef10..4db31bb211 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -113,6 +113,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a7e55caf28..8dd1d30d2a 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -65,6 +65,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index aefc302ed2..8c692f7fb4 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 4871b7ff4d..e5fa4c7864 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 01539b6bd6..1047a35ade 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -730,6 +731,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 5254bc7ef5..8fde56fe60 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -132,6 +132,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
+ amroutine->amskip = btskip;
amroutine->amcostestimate = btcostestimate;
amroutine->amoptions = btoptions;
amroutine->amproperty = btproperty;
@@ -381,6 +382,8 @@ btbeginscan(Relation rel, int nkeys, int norderbys)
*/
so->currTuples = so->markTuples = NULL;
+ so->skipScanKey = NULL;
+
scan->xs_itupdesc = RelationGetDescr(rel);
scan->opaque = so;
@@ -448,6 +451,16 @@ btrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys,
_bt_preprocess_array_keys(scan);
}
+/*
+ * btskip() -- skip to the beginning of the next key prefix
+ */
+bool
+btskip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix)
+{
+ return _bt_skip(scan, direction, indexdir, start, prefix);
+}
+
/*
* btendscan() -- close down a scan
*/
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index c573814f01..4189730f3a 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -37,6 +37,10 @@ static bool _bt_parallel_readpage(IndexScanDesc scan, BlockNumber blkno,
static Buffer _bt_walk_left(Relation rel, Buffer buf, Snapshot snapshot);
static bool _bt_endpoint(IndexScanDesc scan, ScanDirection dir);
static inline void _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir);
+static inline void _bt_update_skip_scankeys(IndexScanDesc scan,
+ Relation indexRel);
+static inline bool _bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir);
/*
@@ -1375,6 +1379,312 @@ _bt_next(IndexScanDesc scan, ScanDirection dir)
return true;
}
+/*
+ * _bt_skip() -- Skip items that have the same prefix as the most recently
+ * fetched index tuple.
+ *
+ * The current position is set so that a subsequent call to _bt_next will
+ * fetch the first tuple that differs in the leading 'prefix' keys.
+ *
+ * There are four different kinds of skipping (depending on dir and
+ * indexdir, that are important to distinguish, especially in the presense
+ * of an index condition:
+ *
+ * * Advancing forward and reading forward
+ * simple scan
+ *
+ * * Advancing forward and reading backward
+ * scan inside a cursor fetching backward, when skipping is necessary
+ * right from the start
+ *
+ * * Advancing backward and reading forward
+ * scan with order by desc inside a cursor fetching forward, when
+ * skipping is necessary right from the start
+ *
+ * * Advancing backward and reading backward
+ * simple scan with order by desc
+ *
+ * The current page is searched for the next unique value. If none is found
+ * we will do a scan from the root in order to find the next page with
+ * a unique value.
+ */
+bool
+_bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ BTStack stack;
+ Buffer buf;
+ OffsetNumber offnum;
+ BTScanPosItem *currItem;
+ Relation indexRel = scan->indexRelation;
+
+ /* We want to return tuples, and we need a starting point */
+ Assert(scan->xs_want_itup);
+ Assert(scan->xs_itup);
+
+ /* If skipScanKey is NULL then we initialize it with _bt_mkscankey */
+ if (so->skipScanKey == NULL)
+ {
+ so->skipScanKey = _bt_mkscankey(indexRel, scan->xs_itup);
+ so->skipScanKey->keysz = prefix;
+ so->skipScanKey->scantid = NULL;
+ }
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /* Check if the next unique key can be found within the current page */
+ if (BTScanPosIsValid(so->currPos) &&
+ _bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir))
+ {
+ bool keyFound = false;
+
+ LockBuffer(so->currPos.buf, BT_READ);
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, so->currPos.buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(so->currPos.buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /* Now read the data */
+ keyFound = _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (keyFound)
+ {
+ /* set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ return true;
+ }
+ }
+
+ if (BTScanPosIsValid(so->currPos))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ /*
+ * We haven't found scan key within the current page, so let's scan from
+ * the root. Use _bt_search and _bt_binsrch to get the buffer and offset
+ * number
+ */
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* Lock the page for SERIALIZABLE transactions */
+ PredicateLockPage(scan->indexRelation, BufferGetBlockNumber(buf),
+ scan->xs_snapshot);
+
+ /* We know in which direction to look */
+ _bt_initialize_more_data(so, dir);
+
+ /*
+ * Simplest case is when both directions are forward, when we are already
+ * at the next distinct key at the beginning of the series (so everything
+ * else would be done in _bt_readpage)
+ *
+ * The case when both directions are backwards is also simple, but we need
+ * to go one step back, since we need a last element from the previous
+ * series.
+ */
+ if (ScanDirectionIsBackward(dir) && ScanDirectionIsBackward(indexdir))
+ offnum = OffsetNumberPrev(offnum);
+
+ /*
+ * Andvance backward but read forward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can read forward without doing anything else. Otherwise
+ * find previous distinct key and the beginning of it's series and read
+ * forward from there. To do so, go back one step, perform binary search
+ * to find the first item in the series and let _bt_readpage do everything
+ * else.
+ */
+ else if (ScanDirectionIsBackward(dir) && ScanDirectionIsForward(indexdir))
+ {
+ if (!scanstart)
+ {
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /* One step back to find a previous value */
+ _bt_readpage(scan, dir, offnum);
+
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (_bt_next(scan, dir))
+ {
+ _bt_update_skip_scankeys(scan, indexRel);
+
+ /*
+ * And now find the last item from the sequence for the
+ * current, value with the intention do OffsetNumberNext. As a
+ * result we end up on a first element from the sequence.
+ */
+ if (_bt_scankey_within_page(scan, so->skipScanKey,
+ so->currPos.buf, dir))
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ else
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+ }
+ }
+ else
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ }
+
+ /*
+ * Advance forward but read backward. At this moment we are at the next
+ * distinct key at the beginning of the series. In case if scan just
+ * started, we can go one step back and read forward without doing
+ * anything else. Otherwise find the next distinct key and the beginning
+ * of it's series, go one step back and read backward from there.
+ *
+ * An interesting situation can happen if one of distinct keys do not pass
+ * a corresponding index condition at all. In this case reading backward
+ * can lead to a previous distinct key being found, creating a loop. To
+ * avoid that check the value to be returned, and jump one more time if
+ * it's the same as at the beginning.
+ */
+ else if (ScanDirectionIsForward(dir) && ScanDirectionIsBackward(indexdir))
+ {
+ if (scanstart)
+ offnum = OffsetNumberPrev(offnum);
+ else
+ {
+ OffsetNumber nextOffset,
+ startOffset;
+
+ nextOffset = startOffset = ItemPointerGetOffsetNumber(&scan->xs_itup->t_tid);
+
+ while (nextOffset == startOffset)
+ {
+ IndexTuple itup;
+
+ /*
+ * Find a next index tuple to update scan key. It could be at
+ * the end, so check for max offset
+ */
+ OffsetNumber curOffnum = offnum;
+ Page page = BufferGetPage(so->currPos.buf);
+ OffsetNumber maxoff = PageGetMaxOffsetNumber(page);
+ ItemId itemid = PageGetItemId(page, Min(offnum, maxoff));
+
+ scan->xs_itup = (IndexTuple) PageGetItem(page, itemid);
+ so->skipScanKey->nextkey = ScanDirectionIsForward(dir);
+
+ _bt_update_skip_scankeys(scan, indexRel);
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ if (BTScanPosIsValid(so->currPos))
+ {
+ ReleaseBuffer(so->currPos.buf);
+ so->currPos.buf = InvalidBuffer;
+ }
+
+ stack = _bt_search(scan->indexRelation, so->skipScanKey,
+ &buf, BT_READ, scan->xs_snapshot);
+ _bt_freestack(stack);
+ so->currPos.buf = buf;
+ offnum = _bt_binsrch(scan->indexRelation, so->skipScanKey, buf);
+
+ /*
+ * Jump to the next key returned the same offset, which means
+ * we are at the end and need to return
+ */
+ if (offnum == curOffnum)
+ {
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+
+ BTScanPosUnpinIfPinned(so->currPos);
+ BTScanPosInvalidate(so->currPos)
+
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+
+ offnum = OffsetNumberPrev(offnum);
+
+ /* Check if _bt_readpage returns already found item */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to
+ * advance to the next page. Return false if there's no
+ * matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+
+ currItem = &so->currPos.items[so->currPos.lastItem];
+ itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+ nextOffset = ItemPointerGetOffsetNumber(&itup->t_tid);
+
+ /*
+ * If the nextOffset is the same as before, it means we are in
+ * the loop, return offnum to the original position and jump
+ * further
+ */
+ if (nextOffset == startOffset)
+ offnum = OffsetNumberNext(offnum);
+ }
+ }
+ }
+
+ /* Now read the data */
+ if (!_bt_readpage(scan, indexdir, offnum))
+ {
+ /*
+ * There's no actually-matching data on this page. Try to advance to
+ * the next page. Return false if there's no matching data at all.
+ */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ if (!_bt_steppage(scan, dir))
+ {
+ pfree(so->skipScanKey);
+ so->skipScanKey = NULL;
+ return false;
+ }
+ }
+ else
+ {
+ /* Drop the lock, and maybe the pin, on the current page */
+ LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK);
+ }
+
+ /* And set IndexTuple */
+ currItem = &so->currPos.items[so->currPos.itemIndex];
+ scan->xs_heaptid = currItem->heapTid;
+ scan->xs_itup = (IndexTuple) (so->currTuples + currItem->tupleOffset);
+
+ return true;
+}
+
/*
* _bt_readpage() -- Load data from current index page into so->currPos
*
@@ -2246,3 +2556,59 @@ _bt_initialize_more_data(BTScanOpaque so, ScanDirection dir)
so->numKilled = 0; /* just paranoia */
so->markItemIndex = -1; /* ditto */
}
+
+/*
+ * _bt_update_skip_scankeys() -- set up a new values for the existing scankeys
+ * based on the current index tuple
+ */
+static inline void
+_bt_update_skip_scankeys(IndexScanDesc scan, Relation indexRel)
+{
+ TupleDesc itupdesc;
+ int indnkeyatts,
+ i;
+ BTScanOpaque so = (BTScanOpaque) scan->opaque;
+ ScanKey scankeys = so->skipScanKey->scankeys;
+
+ itupdesc = RelationGetDescr(indexRel);
+ indnkeyatts = IndexRelationGetNumberOfKeyAttributes(indexRel);
+ for (i = 0; i < indnkeyatts; i++)
+ {
+ Datum datum;
+ bool null;
+ int flags;
+
+ datum = index_getattr(scan->xs_itup, i + 1, itupdesc, &null);
+ flags = (null ? SK_ISNULL : 0) |
+ (indexRel->rd_indoption[i] << SK_BT_INDOPTION_SHIFT);
+ scankeys[i].sk_flags = flags;
+ scankeys[i].sk_argument = datum;
+ }
+}
+
+/*
+ * _bt_scankey_within_page() -- check if the provided scankey could be found
+ * within a page, specified by the buffer.
+ */
+static inline bool
+_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key,
+ Buffer buf, ScanDirection dir)
+{
+ OffsetNumber low,
+ high,
+ compare_offset;
+ Page page = BufferGetPage(buf);
+ BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ int compare_value = ScanDirectionIsForward(dir) ? 0 : 1;
+
+ low = P_FIRSTDATAKEY(opaque);
+ high = PageGetMaxOffsetNumber(page);
+
+ if (unlikely(high < low))
+ return false;
+
+ compare_offset = ScanDirectionIsForward(dir) ? high : low;
+
+ return _bt_compare(scan->indexRelation,
+ key, page, compare_offset) > compare_value;
+}
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4924ae1c59..fa09a4685e 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -68,6 +68,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d573..60cd801247 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -130,6 +130,7 @@ static void ExplainDummyGroup(const char *objtype, const char *labelname,
static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1058,6 +1059,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1380,6 +1397,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1390,6 +1409,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1599,6 +1620,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1612,6 +1635,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..76330f7906 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -65,6 +65,13 @@ IndexOnlyNext(IndexOnlyScanState *node)
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +79,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,14 +122,50 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, direction)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* We can skip the heap fetch if the TID references a heap page on
@@ -250,6 +293,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +549,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 54ad62bb7f..e0cfd710c4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 16083e7a7e..5f723cda4b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 551ce6c41c..028d03a56d 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1820,6 +1820,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1839,6 +1840,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index b5a0033721..710edf160a 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -124,6 +124,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index dff826a828..7b32f2cc7e 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -175,12 +175,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2910,7 +2912,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2921,7 +2924,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5184,7 +5188,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5201,6 +5206,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5213,7 +5219,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5228,6 +5235,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 984fca0696..c84388e6f7 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4834,6 +4834,82 @@ create_distinct_paths(PlannerInfo *root,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+
+ /* Consider index skip scan as well */
+ if (enable_indexskipscan &&
+ IsA(path, IndexPath) &&
+ ((IndexPath *) path)->indexinfo->amcanskip &&
+ root->distinct_pathkeys != NIL)
+ {
+ ListCell *lc;
+ IndexOptInfo *index = NULL;
+ bool different_columns_order = false,
+ not_empty_qual = false;
+ int i = 0;
+ int distinctPrefixKeys;
+
+ Assert(path->pathtype == T_IndexOnlyScan ||
+ path->pathtype == T_IndexScan);
+
+ index = ((IndexPath *) path)->indexinfo;
+ distinctPrefixKeys = list_length(root->query_uniquekeys);
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniquekey = (UniqueKey *) lfirst(lc);
+ EquivalenceMember *em =
+ lfirst_node(EquivalenceMember,
+ list_head(uniquekey->eq_clause->ec_members));
+ Var *var = (Var *) em->em_expr;
+
+ Assert(i < index->ncolumns);
+
+ for (i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ /*
+ * XXX: In case of index scan quals evaluation happens
+ * after ExecScanFetch, which means skip results could be
+ * fitered out. Consider the following query:
+ *
+ * select distinct (a, b) a, b, c from t where c < 100;
+ *
+ * Skip scan returns one tuple for one distinct set of (a,
+ * b) with arbitrary one of c, so if the choosed c does
+ * not match the qual and there is any c that matches the
+ * qual, we miss that tuple.
+ */
+ if (path->pathtype == T_IndexScan &&
+ parse->jointree != NULL &&
+ parse->jointree->quals != NULL &&
+ list_length((List *) parse->jointree->quals) != 0)
+ not_empty_qual = true;
+
+ if (!different_columns_order && !not_empty_qual)
+ {
+ add_path(distinct_rel, (Path *)
+ create_skipscan_unique_path(root,
+ distinct_rel,
+ path,
+ distinctPrefixKeys,
+ numDistinctRows));
+ }
+ }
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index a006dbbe9c..2fb18fb372 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2915,6 +2915,46 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *basepath,
+ int distinctPrefixKeys,
+ double numGroups)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /* The size of the prefix we'll use for skipping. */
+ Assert(pathnode->indexinfo->amcanskip);
+ Assert(distinctPrefixKeys > 0);
+ /*Assert(distinctPrefixKeys <= list_length(pathnode->path.pathkeys));*/
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ /*
+ * The cost to skip to each distinct value should be roughly the same as
+ * the cost of finding the first key times the number of distinct values
+ * we expect to find.
+ */
+ pathnode->path.startup_cost = basepath->startup_cost;
+ pathnode->path.total_cost = basepath->startup_cost * numGroups;
+ pathnode->path.rows = numGroups;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index d82fc5ab8b..f65b299f37 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -271,6 +271,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index e44f71e991..884cdec916 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -922,6 +922,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e1048c0047..a002ee2143 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -353,6 +353,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 3b3e22f73d..3d39cd9d07 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -229,6 +236,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 7e9364a50c..815de4e4dd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 20ace69dab..e098c6a1ab 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -662,6 +662,9 @@ typedef struct BTScanOpaqueData
*/
int markItemIndex; /* itemIndex, or -1 if not valid */
+ /* Work space for _bt_skip */
+ BTScanInsert skipScanKey; /* used to control skipping */
+
/* keep these last in struct for efficiency */
BTScanPosData currPos; /* current position data */
BTScanPosData markPos; /* marked position, if any */
@@ -793,6 +796,8 @@ extern OffsetNumber _bt_binsrch_insert(Relation rel, BTInsertState insertstate);
extern int32 _bt_compare(Relation rel, BTScanInsert key, Page page, OffsetNumber offnum);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
+extern bool _bt_skip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
Snapshot snapshot);
@@ -817,6 +822,8 @@ extern void _bt_end_vacuum_callback(int code, Datum arg);
extern Size BTreeShmemSize(void);
extern void BTreeShmemInit(void);
extern bytea *btoptions(Datum reloptions, bool validate);
+extern bool btskip(IndexScanDesc scan, ScanDirection dir,
+ ScanDirection indexdir, bool start, int prefix);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 1f6f5bbc20..2c6acc160a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1423,6 +1423,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1452,6 +1454,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1470,6 +1474,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 4e329f0fb5..b0ff9ca3a8 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -839,6 +839,7 @@ struct IndexOptInfo
bool amsearchnulls; /* can AM search for NULL/NOT NULL entries? */
bool amhasgettuple; /* does AM have amgettuple interface? */
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
+ bool amcanskip; /* can AM skip duplicate values? */
bool amcanparallel; /* does AM support parallel scan? */
/* Rather than include amapi.h here, we declare amcostestimate like this */
void (*amcostestimate) (); /* AM's cost estimator */
@@ -1189,6 +1190,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1201,6 +1205,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 32c0d87f80..03a00e8e1d 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index cb012ba198..847f34f02b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index f75ff6f323..6c8c9dadbb 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -201,6 +201,11 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ Path *subpath,
+ int numCols,
+ double numGroups);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/test/regress/expected/select_distinct.out b/src/test/regress/expected/select_distinct.out
index f3696c6d1d..51e12ac925 100644
--- a/src/test/regress/expected/select_distinct.out
+++ b/src/test/regress/expected/select_distinct.out
@@ -244,3 +244,508 @@ SELECT null IS NOT DISTINCT FROM null as "yes";
t
(1 row)
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+SELECT DISTINCT a FROM distinct_a;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+ a
+---
+ 1
+(1 row)
+
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+ QUERY PLAN
+--------------------------------------------------------
+ Index Only Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ a | b
+---+---
+ 1 | 2
+ 2 | 2
+ 3 | 2
+ 4 | 2
+ 5 | 2
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+ QUERY PLAN
+----------------------------------------------------
+ Index Only Scan using distinct_a_b_a on distinct_a
+ Skip scan: true
+ Index Cond: (b = 2)
+(3 rows)
+
+DROP INDEX distinct_a_b_a;
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+FETCH FROM c;
+ a | b
+---+---
+ 1 | 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+---
+ 5 | 1
+ 4 | 1
+ 3 | 1
+ 2 | 1
+ 1 | 1
+(5 rows)
+
+END;
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+FETCH FROM c;
+ a | b
+---+-------
+ 5 | 10000
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a | b
+---+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+FETCH 6 FROM c;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a | b
+---+-------
+ 1 | 10000
+ 2 | 10000
+ 3 | 10000
+ 4 | 10000
+ 5 | 10000
+(5 rows)
+
+END;
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+ QUERY PLAN
+--------------------------------------------------------------
+ Index Only Scan using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 1 | 2
+ 3 | 1 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 1 | 2
+ 1 | 1 | 2
+(2 rows)
+
+END;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+ QUERY PLAN
+-----------------------------------------------------------------------
+ Index Only Scan Backward using distinct_abc_a_b_c_idx on distinct_abc
+ Skip scan: true
+ Index Cond: (c = 2)
+(3 rows)
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+FETCH ALL FROM c;
+ a | b | c
+---+---+---
+ 3 | 2 | 2
+ 1 | 2 | 2
+(2 rows)
+
+FETCH BACKWARD ALL FROM c;
+ a | b | c
+---+---+---
+ 1 | 2 | 2
+ 3 | 2 | 2
+(2 rows)
+
+END;
+DROP TABLE distinct_abc;
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+ 2 | 1 | 10
+ 3 | 1 | 10
+ 4 | 1 | 10
+ 5 | 1 | 10
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ a | b | c
+---+---+----
+ 1 | 1 | 10
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+ QUERY PLAN
+---------------------------------------------------
+ Index Scan using distinct_a_a_b_idx on distinct_a
+ Skip scan: true
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+ QUERY PLAN
+-----------------------------------------------------
+ Unique
+ -> Bitmap Heap Scan on distinct_a
+ Recheck Cond: (a = 1)
+ -> Bitmap Index Scan on distinct_a_a_b_idx
+ Index Cond: (a = 1)
+(5 rows)
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+ QUERY PLAN
+---------------------------------------------------------
+ Unique
+ -> Index Scan using distinct_a_a_b_idx on distinct_a
+ Index Cond: (b = 2)
+ Filter: (c = 10)
+(4 rows)
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+ a | a
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 3
+ 4 | 4
+ 5 | 5
+(5 rows)
+
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+ a | ?column?
+---+----------
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+FETCH FROM c;
+ a
+---
+ 1
+(1 row)
+
+FETCH BACKWARD FROM c;
+ a
+---
+(0 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+FETCH 6 FROM c;
+ a
+---
+ 1
+ 2
+ 3
+ 4
+ 5
+(5 rows)
+
+FETCH BACKWARD 6 FROM c;
+ a
+---
+ 5
+ 4
+ 3
+ 2
+ 1
+(5 rows)
+
+END;
+DROP TABLE distinct_a;
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 1
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+ a | b
+---+---
+ 1 | 1
+ 2 | 2
+ 3 | 1
+ 4 | 1
+ 5 | 1
+(5 rows)
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 10000
+ 1 | 10000
+(5 rows)
+
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+ a | b
+---+-------
+ 5 | 10000
+ 4 | 10000
+ 3 | 10000
+ 2 | 9999
+ 1 | 10000
+(5 rows)
+
+DROP TABLE distinct_visibility;
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Index Only Scan using distinct_boundaries_a_b_c_idx on distinct_boundaries
+ Skip scan: true
+ Index Cond: ((b >= 1) AND (c = 0))
+(3 rows)
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+ a | b | c
+---+---+---
+ 1 | 2 | 0
+ 2 | 2 | 0
+ 3 | 2 | 0
+ 4 | 2 | 0
+ 5 | 2 | 0
+(5 rows)
+
+DROP TABLE distinct_boundaries;
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a1c90eb905..bd3b373515 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -78,6 +78,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_hashjoin | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -89,7 +90,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(17 rows)
+(18 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/sql/select_distinct.sql b/src/test/regress/sql/select_distinct.sql
index a605e86449..4c8a50d153 100644
--- a/src/test/regress/sql/select_distinct.sql
+++ b/src/test/regress/sql/select_distinct.sql
@@ -73,3 +73,189 @@ SELECT 1 IS NOT DISTINCT FROM 2 as "no";
SELECT 2 IS NOT DISTINCT FROM 2 as "yes";
SELECT 2 IS NOT DISTINCT FROM null as "no";
SELECT null IS NOT DISTINCT FROM null as "yes";
+
+-- index only skip scan
+CREATE TABLE distinct_a (a int, b int, c int);
+INSERT INTO distinct_a (
+ SELECT five, tenthous, 10 FROM
+ generate_series(1, 5) five,
+ generate_series(1, 10000) tenthous
+);
+CREATE INDEX ON distinct_a (a, b);
+ANALYZE distinct_a;
+
+SELECT DISTINCT a FROM distinct_a;
+SELECT DISTINCT a FROM distinct_a WHERE a = 1;
+SELECT DISTINCT a FROM distinct_a ORDER BY a DESC;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a;
+
+-- test index skip scan with a condition on a non unique field
+SELECT DISTINCT ON (a) a, b FROM distinct_a WHERE b = 2;
+
+-- test index skip scan backwards
+SELECT DISTINCT ON (a) a, b FROM distinct_a ORDER BY a DESC, b DESC;
+
+-- check colums order
+CREATE INDEX distinct_a_b_a on distinct_a (b, a);
+
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT on (a, b) a, b FROM distinct_a WHERE b = 2;
+
+DROP INDEX distinct_a_b_a;
+
+-- test opposite scan/index directions inside a cursor
+-- forward/backward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a, b;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- backward/forward
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b FROM distinct_a ORDER BY a DESC, b DESC;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+-- test missing values and skipping from the end
+CREATE TABLE distinct_abc(a int, b int, c int);
+CREATE INDEX ON distinct_abc(a, b, c);
+INSERT INTO distinct_abc
+ VALUES (1, 1, 1),
+ (1, 1, 2),
+ (1, 2, 2),
+ (1, 2, 3),
+ (2, 2, 1),
+ (2, 2, 3),
+ (3, 1, 1),
+ (3, 1, 2),
+ (3, 2, 2),
+ (3, 2, 3);
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+BEGIN;
+DECLARE c SCROLL CURSOR FOR
+SELECT DISTINCT ON (a) a,b,c FROM distinct_abc WHERE c = 2
+ORDER BY a DESC, b DESC;
+
+FETCH ALL FROM c;
+FETCH BACKWARD ALL FROM c;
+
+END;
+
+DROP TABLE distinct_abc;
+
+-- index skip scan
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a ORDER BY a;
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c
+FROM distinct_a WHERE a = 1 ORDER BY a;
+
+-- check colums order
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT a FROM distinct_a WHERE b = 2 AND c = 10;
+
+-- check projection case
+SELECT DISTINCT a, a FROM distinct_a WHERE b = 2;
+SELECT DISTINCT a, 1 FROM distinct_a WHERE b = 2;
+
+-- test cursor forward/backward movements
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT DISTINCT a FROM distinct_a;
+
+FETCH FROM c;
+FETCH BACKWARD FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+FETCH 6 FROM c;
+FETCH BACKWARD 6 FROM c;
+
+END;
+
+DROP TABLE distinct_a;
+
+-- test tuples visibility
+CREATE TABLE distinct_visibility (a int, b int);
+INSERT INTO distinct_visibility (select a, b from generate_series(1,5) a, generate_series(1, 10000) b);
+CREATE INDEX ON distinct_visibility (a, b);
+ANALYZE distinct_visibility;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 1;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DELETE FROM distinct_visibility WHERE a = 2 and b = 10000;
+SELECT DISTINCT ON (a) a, b FROM distinct_visibility ORDER BY a DESC, b DESC;
+DROP TABLE distinct_visibility;
+
+-- test page boundaries
+CREATE TABLE distinct_boundaries AS
+ SELECT a, b::int2 b, (b % 2)::int2 c FROM
+ generate_series(1, 5) a,
+ generate_series(1,366) b;
+
+CREATE INDEX ON distinct_boundaries (a, b, c);
+ANALYZE distinct_boundaries;
+
+EXPLAIN (COSTS OFF)
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+SELECT DISTINCT ON (a) a, b, c from distinct_boundaries
+WHERE b >= 1 and c = 0 ORDER BY a, b;
+
+DROP TABLE distinct_boundaries;
--
2.21.1
--------------51B4E6A19513F7FD264CD0FF--
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v35 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++++++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++++++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 96 ++++++++++++++++++
src/backend/optimizer/plan/createplan.c | 20 ++--
src/backend/optimizer/util/pathnode.c | 64 ++++++++++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 ++
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 ++
src/include/nodes/execnodes.h | 6 ++
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/test/regress/expected/sysviews.out | 3 +-
30 files changed, 429 insertions(+), 11 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 37f8d8760a..6844a4d5c8 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -148,6 +148,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index efd7201d61..f1551b5f9e 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1702,6 +1723,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1715,6 +1738,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d8cf87e6d0..20a921a700 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index e2f177515d..58c2b64ed3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 42050ab719..17a3e5a722 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1821,6 +1821,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1840,6 +1841,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index b976afb69d..d7d3f7e9fa 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 2a50272da6..340bbfcbb3 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -790,6 +790,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -878,6 +888,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -1027,6 +1039,60 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (can_skip && root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1050,6 +1116,21 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *ukey = lfirst_node(UniqueKey, lc);
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath,
+ ukey->exprs));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1105,6 +1186,21 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *ukey = lfirst_node(UniqueKey, lc);
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath,
+ ukey->exprs));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index eb9543f6ad..5b591036fb 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -3005,7 +3007,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3016,7 +3019,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5283,7 +5287,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5300,6 +5305,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5312,7 +5318,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5327,6 +5334,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 60656bdca5..cac2ca0b0f 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2971,6 +2971,70 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, List *unique_exprs)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ int distinctPrefixKeys;
+ ListCell *lc;
+ List *exprs = NIL;
+
+
+ distinctPrefixKeys = list_length(unique_exprs);
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ /*
+ * Normally we can think about distinctPrefixKeys as just
+ * a number of distinct keys. But if lets say we have a
+ * distinct key a, and the index contains b, a in exactly
+ * this order. In such situation we need to use position
+ * of a in the index as distinctPrefixKeys, otherwise skip
+ * will happen only by the first column.
+ */
+ foreach(lc, unique_exprs)
+ {
+ Expr *unique_expr = (Expr *) lfirst(lc);
+ Var *var = (Var *) unique_expr;
+
+ exprs = lappend(exprs, unique_expr);
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ distinctPrefixKeys = Max(i + 1, distinctPrefixKeys);
+ break;
+ }
+ }
+ }
+
+ Assert(distinctPrefixKeys > 0);
+ pathnode->indexskipprefix = distinctPrefixKeys;
+
+ numDistinctRows = estimate_num_groups(root, exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 2f3e0a70e0..8da64916d3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -961,6 +961,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac02bd0c00..ab7f086d52 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -355,6 +355,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 98e0072b8a..d4bf0b7c34 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 455d9d64ba..95ef75a018 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1232,6 +1232,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1244,6 +1247,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 9710e5c0a4..6e24e520f8 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..705deaf7bd 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ List *unique_exprs);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index a126f0ad61..7e3a36a1b9 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -81,6 +81,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incrementalsort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -92,7 +93,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(20 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--tbesu4k6dthoevcy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v35-0003-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH 07/10] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 9d832ddc03..78500efd7d 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 13905e6037..2b6c007398 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 8830c8df99..8a207332fc 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1872,6 +1872,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1891,6 +1892,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index cd703e41ba..bac3cee9d2 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0008-review-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0008-review-20210317.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v37 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 545 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 26b9927c3a..9f41527bdb 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 649020b7da..1988192fbe 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 1f72562c60..622316ce40 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index ef9b56fd36..a32301e19e 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 25b42e38f2..e909409187 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 7c9ccf446c..d81a3de74b 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3fb8688f8f..8e0f1b0140 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -737,6 +738,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 64d3ba8288..70a610a030 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 43f9b01e83..aa960d9ffb 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1081,6 +1082,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1418,6 +1435,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1428,6 +1447,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1688,6 +1709,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1701,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..670bf1a15e 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 34392f5553..5bd1e1ae5d 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -490,6 +490,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -515,6 +516,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 53cf4fcfa1..282a00089d 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -559,6 +559,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -573,6 +574,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 7b9e8c3292..58fb93aada 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1826,6 +1826,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1845,6 +1846,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 733f7ea543..cf31c42552 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 94280a730c..d62e96c24c 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index abcfe4d9d7..0e682920f6 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3004,6 +3004,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index bdd4876a84..6fd82ea6fd 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a62d64eaa4..005de5fb56 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9cb571f7cc..d5d7b7834d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 85b4766016..9f0e81ec94 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -172,6 +172,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -274,6 +280,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 68d90f5141..f3c5668dd0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6c0a7d68d6..0b0464c0b1 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1425,6 +1425,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1454,6 +1456,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1472,6 +1476,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 4a4ae3e926..bbab87411f 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1228,6 +1228,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1240,6 +1243,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 7e6b10f86b..aed2ceacd0 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -401,6 +401,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -428,6 +430,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 6141654e47..67192a9188 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 81bdacf59d..5cbc2c49cf 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -95,6 +95,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -106,7 +107,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--snesplnlompos6d4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v37-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH 07/10] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 9d832ddc03..78500efd7d 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 13905e6037..2b6c007398 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 8830c8df99..8a207332fc 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1872,6 +1872,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1891,6 +1892,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index cd703e41ba..bac3cee9d2 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0008-review-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0008-review-20210317.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v38 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 ++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 196 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 7 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 554 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 1e505b1da5..a58bdf7604 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index ec5741df6d..3442ae816b 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 27ba596c6e..1ec10ec513 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 6b9b04cf42..d776d2732f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..2544ea24f1 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index d8b1815061..b2ed0712f2 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index afc45429ba..a160de5493 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -149,6 +149,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1098,6 +1099,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1439,6 +1456,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1449,6 +1468,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1709,6 +1730,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1722,6 +1745,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..0fad258202 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * If we already emitted first tuple, while doing index only skip scan
+ * with advancing and reading in different directions we can return to
+ * the same position where we started after visibility check. Recognize
+ * such situations and skip more.
+ */
+ if ((readDirection != direction) && node->ioss_FirstTupleEmitted &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 2fffb1b437..71aac4493d 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..481dbf2e00 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -491,6 +491,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -516,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..eb51324c61 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -560,6 +560,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -574,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..a23a405523 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1870,6 +1870,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1889,6 +1890,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index a25b674a19..658bacabf4 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -132,6 +132,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index ff536e6b24..bd0a073998 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,129 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1181,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1245,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1263,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 4c90f63705..e15637b514 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 906cab7053..c2969a0279 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -181,12 +181,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2997,7 +2999,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -3008,7 +3011,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5340,7 +5344,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5357,6 +5362,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5369,7 +5375,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5384,6 +5391,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index c995921c88..a768f43e22 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3029,6 +3029,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cb31b64047 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -281,6 +281,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 855076b1fd..6354958ed3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -960,6 +960,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f46c2dd7a8..4eecf2d95d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -359,6 +359,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..eacf890dbe 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,12 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +281,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..f14baadea0 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e31ad6204e..9982343e8a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1427,6 +1427,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1456,6 +1458,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1474,6 +1478,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 27ff639053..01e68fe6db 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1241,6 +1241,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1253,6 +1256,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 95292d7573..a23ea622ed 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -403,6 +403,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -430,6 +432,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 1be93be098..0f22fcef99 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index aa6c3e439e..872c858a60 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -210,6 +210,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b571ddec11..45a985e8c1 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -205,8 +205,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 6d048e309c..eda1b0a3b4 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -102,6 +102,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -113,7 +114,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.2
--yh7cw345mbttadjz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v38-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v39 2/5] Index skip scan
@ 2021-05-20 19:16 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2021-05-20 19:16 UTC (permalink / raw)
Allow IndexOnlyScan to skip duplicated tuples based on search key prefix
(a trick also known as Index Skip Scan or Loose Index Scan, see in the
wiki [1]). The idea is to avoid scanning all equal values of a key, as
soon as a new value is found, restart the search by looking for a larger
value. This approach is much faster when the index has many equal keys.
Implemented via equipping IndexPath with indexskipprefix field and
creating an extra IndexPath with such prefix if suitable unique
expressions are present. On the execution size a new index am function
amskip is introduced to provide index specific implementation for such
skipping. To simplify potential amskip implementations,
ExecSupportsBackwardScan now returns false in case if index skip scan is
used, otherwise amskip has to deal with scroll cursor and be prepared to
handle different advance/read directions. ExecSupportsBackwardScan may
seem to have too big scope, but looks like now it used only together
with cursorOptions checks for CURSOR_OPT_SCROLL.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 ++
doc/src/sgml/indexam.sgml | 43 ++++++
doc/src/sgml/indices.sgml | 23 +++
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 16 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 23 +++
src/backend/executor/execAmi.c | 32 +++-
src/backend/executor/nodeIndexonlyscan.c | 47 +++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 140 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++++-
src/backend/optimizer/plan/createplan.c | 10 +-
src/backend/optimizer/util/pathnode.c | 37 +++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 6 +
src/include/access/genam.h | 1 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 2 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 2 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
34 files changed, 477 insertions(+), 19 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 754de008d4..f7288cf6dc 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5efbfe97b5..114b4c9b5f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5056,6 +5056,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 29fa3b7933..6c364a53d8 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -151,6 +151,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -768,6 +769,48 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index 56fbd45178..b0c8744205 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1297,6 +1297,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index c23ea44866..f0ce6f7334 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -118,6 +118,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index cdd626ff0a..aeb1d40b0f 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..fed061184e 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0752fb38a9..fd7c13ee4c 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 5e22479b7a..310aacca58 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,21 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8d99c9b762..f9b1e395f4 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -71,6 +71,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 8ab7bca866..37ca8612f4 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -152,6 +152,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1118,6 +1119,21 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1475,6 +1491,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1748,6 +1765,12 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ {
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainPropertyBool("Skip scan", true, es);
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+ }
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index b3726a54f3..4248912239 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -64,7 +64,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
-static bool IndexSupportsBackwardScan(Oid indexid);
+static bool IndexSupportsBackwardScan(Plan *node);
/*
@@ -557,10 +557,8 @@ ExecSupportsBackwardScan(Plan *node)
return false;
case T_IndexScan:
- return IndexSupportsBackwardScan(((IndexScan *) node)->indexid);
-
case T_IndexOnlyScan:
- return IndexSupportsBackwardScan(((IndexOnlyScan *) node)->indexid);
+ return IndexSupportsBackwardScan(node);
case T_SubqueryScan:
return ExecSupportsBackwardScan(((SubqueryScan *) node)->subplan);
@@ -604,16 +602,38 @@ ExecSupportsBackwardScan(Plan *node)
/*
* An IndexScan or IndexOnlyScan node supports backward scan only if the
- * index's AM does.
+ * index's AM does and no skip scan is used.
*/
static bool
-IndexSupportsBackwardScan(Oid indexid)
+IndexSupportsBackwardScan(Plan *node)
{
bool result;
+ Oid indexid = InvalidOid;
+ int skip_prefix_size = 0;
HeapTuple ht_idxrel;
Form_pg_class idxrelrec;
IndexAmRoutine *amroutine;
+ Assert(IsA(node, IndexScan) || IsA(node, IndexOnlyScan));
+ switch(nodeTag(node))
+ {
+ case T_IndexScan:
+ indexid = ((IndexScan *) node)->indexid;
+ break;
+
+ case T_IndexOnlyScan:
+ indexid = ((IndexOnlyScan *) node)->indexid;
+ skip_prefix_size = ((IndexOnlyScan *) node)->indexskipprefixsize;
+ break;
+
+ default:
+ elog(DEBUG2, "unrecognized node type: %d", (int) nodeTag(node));
+ break;
+ }
+
+ if (skip_prefix_size > 0)
+ return false;
+
/* Fetch the pg_class tuple of the index relation */
ht_idxrel = SearchSysCache1(RELOID, ObjectIdGetDatum(indexid));
if (!HeapTupleIsValid(ht_idxrel))
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 0754e28a9a..e800e43e35 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,17 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
- ItemPointer tid;
+ ItemPointer tid = NULL;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +81,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,15 +124,43 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix.
+ */
+ if (node->ioss_SkipPrefixSize > 0 && node->ioss_FirstTupleEmitted)
+ {
+ if (!index_skip(scandesc, direction, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +287,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +543,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index f5a7760740..320d648540 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -517,6 +517,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c723f6d635..a6daf3847a 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -575,6 +575,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3746668f52..ab5f237305 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1887,6 +1887,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 8577c7b138..d449876855 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -133,6 +133,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 0e4e00eaf0..2622f6e389 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,72 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip & index_only_scan;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1124,33 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /*
+ * All distinct keys are constant and optimized away.
+ * Skipping with 1 is sufficient.
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1189,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1207,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index f28547148d..154cb30745 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -523,6 +523,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -535,7 +576,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -546,7 +589,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -555,6 +599,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -608,7 +654,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index a9aff24831..513466f7b6 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -190,7 +190,8 @@ static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -3067,7 +3068,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -5439,7 +5441,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5454,6 +5457,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index e190b20675..0b24f670b4 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3080,6 +3080,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ List *uniqExprs;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ uniqExprs = linitial_node(List, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, uniqExprs,
+ pathnode->path.rows,
+ NULL, NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index c5194fdbbf..df20b88316 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -271,6 +271,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index f2c7c2486b..f0671e406c 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -985,6 +985,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0f7f49b949..0be508ef04 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -365,6 +365,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..b1e82f4fb5 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,11 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -275,6 +280,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 480a4762f5..e757f01b0d 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,7 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *istat);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 8154adf3b8..6690c2a0b0 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index e7ae21c023..c480794f59 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1517,6 +1517,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 6c04ba2bb2..d53dda7ef7 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1234,6 +1234,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1246,6 +1249,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index d671328dfd..530e9ab875 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -434,6 +434,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 0fe60d82e4..baa8655c55 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index b976f2963e..e9e56873a6 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -217,6 +217,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index b613392e2f..d4288f8451 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -212,8 +212,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 0bb558d93c..fa54f00ae7 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -103,6 +103,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -115,7 +116,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(20 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.26.3
--mlmus2qy2pkmy4td
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v39-0003-amskip-implementation-for-Btree.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v40 2/6] Index skip scan
@ 2022-01-08 16:16 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2022-01-08 16:16 UTC (permalink / raw)
Allow IndexOnlyScan to skip duplicated tuples based on search key prefix
(a trick also known as Index Skip Scan or Loose Index Scan, see in the
wiki [1]). The idea is to avoid scanning all equal values of a key, as
soon as a new value is found, restart the search by looking for a larger
value. This approach is much faster when the index has many equal keys.
Implemented via equipping IndexPath with indexskipprefix field and
creating an extra IndexPath with such prefix if suitable unique
expressions are present. On the execution size a new index am function
amskip is introduced to provide index specific implementation for such
skipping. To simplify potential amskip implementations,
ExecSupportsBackwardScan now returns false in case if index skip scan is
used, otherwise amskip has to deal with scroll cursor and be prepared to
handle different advance/read directions. ExecSupportsBackwardScan may
seem to have too big scope, but looks like now it used only together
with cursorOptions checks for CURSOR_OPT_SCROLL.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 ++
doc/src/sgml/indexam.sgml | 43 ++++++
doc/src/sgml/indices.sgml | 23 +++
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 16 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 23 +++
src/backend/executor/execAmi.c | 32 +++-
src/backend/executor/nodeIndexonlyscan.c | 47 +++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 140 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++++-
src/backend/optimizer/plan/createplan.c | 10 +-
src/backend/optimizer/util/pathnode.c | 37 +++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 6 +
src/include/access/genam.h | 1 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 2 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 2 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
34 files changed, 477 insertions(+), 19 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index a434cf93ef..3b312c039d 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4cd9818acf..47663eb583 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5012,6 +5012,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 84de931071..ffc884a038 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -153,6 +153,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -779,6 +780,48 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index 023157d888..ab9595d37f 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1297,6 +1297,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index ba78ecff66..d9ee7eea6d 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -119,6 +119,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 3d15701a01..56292eb822 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -67,6 +67,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index c3cdfca9a2..459de6ddc2 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -88,6 +88,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index d48c8a4549..a9c02b6559 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -85,6 +85,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index fe80b8b0ba..bcf7c73467 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,21 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 1ae7492216..0b0dfa278f 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -73,6 +73,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index b970997c34..45e03c5207 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -152,6 +152,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1114,6 +1115,21 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1471,6 +1487,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1744,6 +1761,12 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ {
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainPropertyBool("Skip scan", true, es);
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+ }
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->recheckqual)
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index b6245994f0..ced8933f44 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -64,7 +64,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
-static bool IndexSupportsBackwardScan(Oid indexid);
+static bool IndexSupportsBackwardScan(Plan *node);
/*
@@ -555,10 +555,8 @@ ExecSupportsBackwardScan(Plan *node)
return false;
case T_IndexScan:
- return IndexSupportsBackwardScan(((IndexScan *) node)->indexid);
-
case T_IndexOnlyScan:
- return IndexSupportsBackwardScan(((IndexOnlyScan *) node)->indexid);
+ return IndexSupportsBackwardScan(node);
case T_SubqueryScan:
return ExecSupportsBackwardScan(((SubqueryScan *) node)->subplan);
@@ -598,16 +596,38 @@ ExecSupportsBackwardScan(Plan *node)
/*
* An IndexScan or IndexOnlyScan node supports backward scan only if the
- * index's AM does.
+ * index's AM does and no skip scan is used.
*/
static bool
-IndexSupportsBackwardScan(Oid indexid)
+IndexSupportsBackwardScan(Plan *node)
{
bool result;
+ Oid indexid = InvalidOid;
+ int skip_prefix_size = 0;
HeapTuple ht_idxrel;
Form_pg_class idxrelrec;
IndexAmRoutine *amroutine;
+ Assert(IsA(node, IndexScan) || IsA(node, IndexOnlyScan));
+ switch(nodeTag(node))
+ {
+ case T_IndexScan:
+ indexid = ((IndexScan *) node)->indexid;
+ break;
+
+ case T_IndexOnlyScan:
+ indexid = ((IndexOnlyScan *) node)->indexid;
+ skip_prefix_size = ((IndexOnlyScan *) node)->indexskipprefixsize;
+ break;
+
+ default:
+ elog(DEBUG2, "unrecognized node type: %d", (int) nodeTag(node));
+ break;
+ }
+
+ if (skip_prefix_size > 0)
+ return false;
+
/* Fetch the pg_class tuple of the index relation */
ht_idxrel = SearchSysCache1(RELOID, ObjectIdGetDatum(indexid));
if (!HeapTupleIsValid(ht_idxrel))
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index eb3ddd2943..40ad1b949b 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,17 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
- ItemPointer tid;
+ ItemPointer tid = NULL;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +81,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,15 +124,43 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix.
+ */
+ if (node->ioss_SkipPrefixSize > 0 && node->ioss_FirstTupleEmitted)
+ {
+ if (!index_skip(scandesc, direction, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -248,6 +285,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -502,6 +541,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 90b5da51c9..af8678ce51 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -523,6 +523,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 2b0236937a..473d522db3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -584,6 +584,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3f68f7c18d..169fb408d3 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1888,6 +1888,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 8dc7dd4ca2..efb2954338 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -133,6 +133,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 0ef70ad7f1..00526f3476 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,72 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip & index_only_scan;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1124,33 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /*
+ * All distinct keys are constant and optimized away.
+ * Skipping with 1 is sufficient.
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1189,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1207,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e2be1fbf90..cfdff4eee9 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -523,6 +523,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -535,7 +576,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -546,7 +589,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -555,6 +599,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -608,7 +654,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index cd6d72c763..98738c9d9c 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -191,7 +191,8 @@ static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
List *indexqual, List *recheckqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -3092,7 +3093,8 @@ create_indexscan_plan(PlannerInfo *root,
stripped_indexquals,
fixed_indexorderbys,
indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -5466,7 +5468,8 @@ make_indexonlyscan(List *qptlist,
List *recheckqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5482,6 +5485,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index abb77d867e..0c61795389 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3095,6 +3095,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ List *uniqExprs;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ uniqExprs = linitial_node(List, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, uniqExprs,
+ pathnode->path.rows,
+ NULL, NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index a5002ad895..1e6fb0c543 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 4c94f09c64..6a1a2e42a8 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -1002,6 +1002,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 817d5f5324..ac9abfb552 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -367,6 +367,7 @@
#enable_incremental_sort = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_memoize = on
#enable_mergejoin = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index a382551a98..cb2f48a1bc 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,11 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -277,6 +282,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 134b20f1e6..d13d95c458 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,7 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *istat);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 1ab4d5e19a..fd71629da4 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 4ea8735dd8..f28ec2f830 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1526,6 +1526,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 056b13826a..40997ee759 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1243,6 +1243,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1255,6 +1258,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 0b518ce6b2..6b3eefebc6 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -453,6 +453,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 356a51f370..03d5816c82 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index bb6d730e93..227cda4bd7 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -218,6 +218,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 3dfa21adad..72b3bd9059 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -212,8 +212,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 2088857615..cd564b0316 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -110,6 +110,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_memoize | on
enable_mergejoin | on
@@ -122,7 +123,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(20 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.32.0
--hfgz4nnxbxkjm2qw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v40-0003-amskip-implementation-for-Btree.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* [PATCH v41 2/6] Index skip scan
@ 2022-01-08 16:16 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Dmitrii Dolgov @ 2022-01-08 16:16 UTC (permalink / raw)
Allow IndexOnlyScan to skip duplicated tuples based on search key prefix
(a trick also known as Index Skip Scan or Loose Index Scan, see in the
wiki [1]). The idea is to avoid scanning all equal values of a key, as
soon as a new value is found, restart the search by looking for a larger
value. This approach is much faster when the index has many equal keys.
Implemented via equipping IndexPath with indexskipprefix field and
creating an extra IndexPath with such prefix if suitable unique
expressions are present. On the execution size a new index am function
amskip is introduced to provide index specific implementation for such
skipping. To simplify potential amskip implementations,
ExecSupportsBackwardScan now returns false in case if index skip scan is
used, otherwise amskip has to deal with scroll cursor and be prepared to
handle different advance/read directions. ExecSupportsBackwardScan may
seem to have too big scope, but looks like now it used only together
with cursorOptions checks for CURSOR_OPT_SCROLL.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/config.sgml | 15 ++
doc/src/sgml/indexam.sgml | 43 ++++++
doc/src/sgml/indices.sgml | 23 +++
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 16 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 23 +++
src/backend/executor/execAmi.c | 32 +++-
src/backend/executor/nodeIndexonlyscan.c | 47 +++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 140 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++++-
src/backend/optimizer/plan/createplan.c | 10 +-
src/backend/optimizer/util/pathnode.c | 37 +++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 10 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 6 +
src/include/access/genam.h | 1 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 2 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 2 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
34 files changed, 478 insertions(+), 19 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index a434cf93ef..3b312c039d 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7a48973b3c..e43295861f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5035,6 +5035,21 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-indexskipscan" xreflabel="enable_indexskipscan">
+ <term><varname>enable_indexskipscan</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_indexskipscan</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables or disables the query planner's use of index-skip-scan plan
+ types (see <xref linkend="indexes-index-skip-scans"/>). The default is
+ <literal>on</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-enable-material" xreflabel="enable_material">
<term><varname>enable_material</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index d4163c96e9..31081d0f8d 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -153,6 +153,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
@@ -779,6 +780,48 @@ amrestrpos (IndexScanDesc scan);
<para>
<programlisting>
+bool
+amskip (IndexScanDesc scan,
+ ScanDirection direction,
+ int prefix);
+</programlisting>
+ Skip past all tuples where the first 'prefix' columns have the same value as
+ the last tuple returned in the current scan. The arguments are:
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>scan</parameter></term>
+ <listitem>
+ <para>
+ Index scan information
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>direction</parameter></term>
+ <listitem>
+ <para>
+ The direction in which data is advancing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><parameter>prefix</parameter></term>
+ <listitem>
+ <para>
+ Distinct prefix size.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </para>
+
+ <para>
+<programlisting>
Size
amestimateparallelscan (void);
</programlisting>
diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml
index 023157d888..ab9595d37f 100644
--- a/doc/src/sgml/indices.sgml
+++ b/doc/src/sgml/indices.sgml
@@ -1297,6 +1297,29 @@ SELECT target FROM tests WHERE subject = 'some-subject' AND success;
and later will recognize such cases and allow index-only scans to be
generated, but older versions will not.
</para>
+
+ <sect2 id="indexes-index-skip-scans">
+ <title>Index Skip Scans</title>
+
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index</primary>
+ <secondary>index-skip scans</secondary>
+ </indexterm>
+ <indexterm zone="indexes-index-skip-scans">
+ <primary>index-skip scan</primary>
+ </indexterm>
+
+ <para>
+ When the rows retrieved from an index scan are then deduplicated by
+ eliminating rows matching on a prefix of index keys (e.g. when using
+ <literal>SELECT DISTINCT</literal>), the planner will consider
+ skipping groups of rows with a matching key prefix. When a row with
+ a particular prefix is found, remaining rows with the same key prefix
+ are skipped. The larger the number of rows with the same key prefix
+ rows (i.e. the lower the number of distinct key prefixes in the index),
+ the more efficient this is.
+ </para>
+ </sect2>
</sect1>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 4366010768..0d04b299d3 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -119,6 +119,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 3d15701a01..56292eb822 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -67,6 +67,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8c6c744ab7..16a45f05bc 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -89,6 +89,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..41e5e9b594 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -86,6 +86,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index fe80b8b0ba..bcf7c73467 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -739,6 +740,21 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 1ae7492216..0b0dfa278f 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -73,6 +73,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 9f632285b6..4f5bd1d678 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -152,6 +152,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1120,6 +1121,21 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1477,6 +1493,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1750,6 +1767,12 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ {
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainPropertyBool("Skip scan", true, es);
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+ }
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->recheckqual)
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index b6245994f0..ced8933f44 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -64,7 +64,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
-static bool IndexSupportsBackwardScan(Oid indexid);
+static bool IndexSupportsBackwardScan(Plan *node);
/*
@@ -555,10 +555,8 @@ ExecSupportsBackwardScan(Plan *node)
return false;
case T_IndexScan:
- return IndexSupportsBackwardScan(((IndexScan *) node)->indexid);
-
case T_IndexOnlyScan:
- return IndexSupportsBackwardScan(((IndexOnlyScan *) node)->indexid);
+ return IndexSupportsBackwardScan(node);
case T_SubqueryScan:
return ExecSupportsBackwardScan(((SubqueryScan *) node)->subplan);
@@ -598,16 +596,38 @@ ExecSupportsBackwardScan(Plan *node)
/*
* An IndexScan or IndexOnlyScan node supports backward scan only if the
- * index's AM does.
+ * index's AM does and no skip scan is used.
*/
static bool
-IndexSupportsBackwardScan(Oid indexid)
+IndexSupportsBackwardScan(Plan *node)
{
bool result;
+ Oid indexid = InvalidOid;
+ int skip_prefix_size = 0;
HeapTuple ht_idxrel;
Form_pg_class idxrelrec;
IndexAmRoutine *amroutine;
+ Assert(IsA(node, IndexScan) || IsA(node, IndexOnlyScan));
+ switch(nodeTag(node))
+ {
+ case T_IndexScan:
+ indexid = ((IndexScan *) node)->indexid;
+ break;
+
+ case T_IndexOnlyScan:
+ indexid = ((IndexOnlyScan *) node)->indexid;
+ skip_prefix_size = ((IndexOnlyScan *) node)->indexskipprefixsize;
+ break;
+
+ default:
+ elog(DEBUG2, "unrecognized node type: %d", (int) nodeTag(node));
+ break;
+ }
+
+ if (skip_prefix_size > 0)
+ return false;
+
/* Fetch the pg_class tuple of the index relation */
ht_idxrel = SearchSysCache1(RELOID, ObjectIdGetDatum(indexid));
if (!HeapTupleIsValid(ht_idxrel))
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index eb3ddd2943..40ad1b949b 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,17 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
- ItemPointer tid;
+ ItemPointer tid = NULL;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -72,7 +81,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -115,15 +124,43 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix.
+ */
+ if (node->ioss_SkipPrefixSize > 0 && node->ioss_FirstTupleEmitted)
+ {
+ if (!index_skip(scandesc, direction, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
+
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -248,6 +285,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -502,6 +541,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2b..fe0d92ad46 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -523,6 +523,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 6bdad462c7..06eb1a89e9 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -584,6 +584,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3f68f7c18d..169fb408d3 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1888,6 +1888,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 4d9f3b4bb6..6c42ae2121 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -133,6 +133,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 0ef70ad7f1..00526f3476 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -784,6 +784,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -866,12 +876,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -989,7 +1002,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1021,6 +1035,72 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip & index_only_scan;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are
+ * not covered by index. The reason for that is that those conditions
+ * are evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *) root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ OpExpr *expr_op;
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+ expr_op = (OpExpr *) qual;
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ /*
+ * Check if the qual operator is indexable by any columns of
+ * the index, test collation and opfamily.
+ */
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno &&
+ IndexCollMatchesExprColl(index->indexcollations[i],
+ expr_op->inputcollid) &&
+ op_in_opfamily(expr_op->opno, index->opfamily[i]))
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1044,6 +1124,33 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /*
+ * All distinct keys are constant and optimized away.
+ * Skipping with 1 is sufficient.
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1082,7 +1189,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1099,6 +1207,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e2be1fbf90..cfdff4eee9 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -523,6 +523,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -535,7 +576,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -546,7 +589,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -555,6 +599,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -608,7 +654,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index fa069a217c..511dda3a9f 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -191,7 +191,8 @@ static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
List *indexqual, List *recheckqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -3108,7 +3109,8 @@ create_indexscan_plan(PlannerInfo *root,
stripped_indexquals,
fixed_indexorderbys,
indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -5482,7 +5484,8 @@ make_indexonlyscan(List *qptlist,
List *recheckqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5498,6 +5501,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index abb77d867e..0c61795389 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3095,6 +3095,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ List *uniqExprs;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ uniqExprs = linitial_node(List, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, uniqExprs,
+ pathnode->path.rows,
+ NULL, NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index a5002ad895..1e6fb0c543 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 932aefc777..6410c6ede7 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -1010,6 +1010,16 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL,
+ GUC_EXPLAIN
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 4cf5b26a36..17ff364a7a 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -370,6 +370,7 @@
#enable_incremental_sort = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_memoize = on
#enable_mergejoin = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index a382551a98..cb2f48a1bc 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -173,6 +173,11 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -277,6 +282,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 134b20f1e6..d13d95c458 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -183,6 +183,7 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *istat);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 1ab4d5e19a..fd71629da4 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 44dd73fc80..cf36e6c0e6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1533,6 +1533,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 056b13826a..40997ee759 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1243,6 +1243,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1255,6 +1258,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 0b518ce6b2..6b3eefebc6 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -453,6 +453,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 356a51f370..03d5816c82 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index bb6d730e93..227cda4bd7 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -218,6 +218,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 3dfa21adad..72b3bd9059 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -212,8 +212,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 442eeb1e3f..1ada679d46 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -110,6 +110,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_memoize | on
enable_mergejoin | on
@@ -122,7 +123,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(20 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.32.0
--3ps4beyi6rmp3n5j
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v41-0003-amskip-implementation-for-Btree.patch"
^ permalink raw reply [nested|flat] 58+ messages in thread
* broken master branch
@ 2023-04-08 08:03 Pavel Stehule <[email protected]>
2023-04-08 08:38 ` Re: broken master branch Thomas Munro <[email protected]>
0 siblings, 1 reply; 58+ messages in thread
From: Pavel Stehule @ 2023-04-08 08:03 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Hi
on fresh Fedora 38, I cannot to run regress tests
+ERROR: could not load library
"/home/pavel/src/postgresql.master/tmp_install/usr/local/pgsql/master/lib/llvmjit.so":
/home/pavel/src/p
ostgresql.master/tmp_install/usr/local/pgsql/master/lib/llvmjit.so:
undefined symbol: LLVMBuildGEP
SELECT BOOLTBL1.*, BOOLTBL2.*
FROM BOOLTBL1, BOOLTBL2
WHERE boolne(BOOLTBL2.f1,BOOLTBL1.f1);
There is lot of compile warnings
In file included from llvmjit_expr.c:31:
../../../../src/include/jit/llvmjit_emit.h: In function ‘l_load_struct_gep’:
../../../../src/include/jit/llvmjit_emit.h:112:30: warning: implicit
declaration of function ‘LLVMBuildStructGEP’; did you mean
‘LLVMBuildStructGEP2’? [-Wimplicit-function-declaration]
112 | LLVMValueRef v_ptr = LLVMBuildStructGEP(b, v, idx, "");
| ^~~~~~~~~~~~~~~~~~
| LLVMBuildStructGEP2
../../../../src/include/jit/llvmjit_emit.h:112:30: warning: initialization
of ‘LLVMValueRef’ {aka ‘struct LLVMOpaqueValue *’} from ‘int’ makes pointer
from integer without a cast [-Wint-conversion]
../../../../src/include/jit/llvmjit_emit.h:114:16: warning: implicit
declaration of function ‘LLVMBuildLoad’; did you mean ‘LLVMBuildLoad2’?
[-Wimplicit-function-declaration]
114 | return LLVMBuildLoad(b, v_ptr, name);
| ^~~~~~~~~~~~~
| LLVMBuildLoad2
../../../../src/include/jit/llvmjit_emit.h:114:16: warning: returning ‘int’
from a function with return type ‘LLVMValueRef’ {aka ‘struct
LLVMOpaqueValue *’} makes pointer from integer without a cast
[-Wint-conversion]
114 | return LLVMBuildLoad(b, v_ptr, name);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/jit/llvmjit_emit.h: In function ‘l_load_gep1’:
../../../../src/include/jit/llvmjit_emit.h:123:30: warning: implicit
declaration of function ‘LLVMBuildGEP’; did you mean ‘LLVMBuildGEP2’?
[-Wimplicit-function-declaration]
123 | LLVMValueRef v_ptr = LLVMBuildGEP(b, v, &idx, 1, "");
| ^~~~~~~~~~~~
| LLVMBuildGEP2
../../../../src/include/jit/llvmjit_emit.h:123:30: warning: initialization
of ‘LLVMValueRef’ {aka ‘struct LLVMOpaqueValue *’} from ‘int’ makes pointer
from integer without a cast [-Wint-conversion]
../../../../src/include/jit/llvmjit_emit.h:125:16: warning: returning ‘int’
from a function with return type ‘LLVMValueRef’ {aka ‘struct
LLVMOpaqueValue *’} makes pointer from integer without a cast
[-Wint-conversion]
Regards
Pavel
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: broken master branch
2023-04-08 08:03 broken master branch Pavel Stehule <[email protected]>
@ 2023-04-08 08:38 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 58+ messages in thread
From: Thomas Munro @ 2023-04-08 08:38 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Sat, Apr 8, 2023 at 8:04 PM Pavel Stehule <[email protected]> wrote:
> on fresh Fedora 38, I cannot to run regress tests
Looks like the new LLVM 16. I'll try to look at this again next week.
In the meantime you could try using 15.
^ permalink raw reply [nested|flat] 58+ messages in thread
end of thread, other threads:[~2023-04-08 08:38 UTC | newest]
Thread overview: 58+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-07-03 14:25 [PATCH v28] Index skip scan erthalion <[email protected]>
2019-11-15 14:46 [PATCH v32 2/2] Index skip scan jesperpedersen <[email protected]>
2019-11-15 14:46 [PATCH v33 2/2] Index skip scan jesperpedersen <[email protected]>
2019-11-15 14:46 [PATCH v31 2/2] Index skip scan jesperpedersen <[email protected]>
2019-11-15 14:46 [PATCH v33 2/2] Index skip scan jesperpedersen <[email protected]>
2020-01-20 13:01 [PATCH 2/2] Index skip scan jesperpedersen <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v35 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH 07/10] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v37 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH 07/10] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2020-06-08 18:39 [PATCH v38 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2021-05-20 19:16 [PATCH v39 2/5] Index skip scan Dmitrii Dolgov <[email protected]>
2022-01-08 16:16 [PATCH v40 2/6] Index skip scan Dmitrii Dolgov <[email protected]>
2022-01-08 16:16 [PATCH v41 2/6] Index skip scan Dmitrii Dolgov <[email protected]>
2023-04-08 08:03 broken master branch Pavel Stehule <[email protected]>
2023-04-08 08:38 ` Re: broken master branch Thomas Munro <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox