public inbox for [email protected]help / color / mirror / Atom feed
Re: index prefetching 10+ messages / 4 participants [nested] [flat]
* Re: index prefetching @ 2023-07-14 20:31 Tomas Vondra <[email protected]> 0 siblings, 2 replies; 10+ messages in thread From: Tomas Vondra @ 2023-07-14 20:31 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> Here's a v5 of the patch, rebased to current master and fixing a couple compiler warnings reported by cfbot (%lu vs. UINT64_FORMAT in some debug messages). No other changes compared to v4. cfbot also reported a failure on windows in pg_dump [1], but it seem pretty strange: [11:42:48.708] ------------------------------------- 8< ------------------------------------- [11:42:48.708] stderr: [11:42:48.708] # Failed test 'connecting to an invalid database: matches' The patch does nothing related to pg_dump, and the test works perfectly fine for me (I don't have windows machine, but 32-bit and 64-bit linux works fine for me). regards [1] https://cirrus-ci.com/task/6398095366291456 -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company Attachments: [text/x-patch] index-prefetch-v5.patch (39.8K, 2-index-prefetch-v5.patch) download | inline diff: diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c index e2c9b5f069..9045c6eb7a 100644 --- a/src/backend/access/gist/gistget.c +++ b/src/backend/access/gist/gistget.c @@ -678,7 +678,6 @@ gistgettuple(IndexScanDesc scan, ScanDirection dir) scan->xs_hitup = so->pageData[so->curPageData].recontup; so->curPageData++; - return true; } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 5a17112c91..0b6c920ebd 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -44,6 +44,7 @@ #include "storage/smgr.h" #include "utils/builtins.h" #include "utils/rel.h" +#include "utils/spccache.h" static void reform_and_rewrite_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, @@ -751,6 +752,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, PROGRESS_CLUSTER_INDEX_RELID }; int64 ci_val[2]; + int prefetch_target; + + prefetch_target = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; @@ -759,7 +763,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tableScan = NULL; heapScan = NULL; - indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0); + indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0, + prefetch_target, prefetch_target); index_rescan(indexScan, NULL, 0, NULL, 0); } else diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 722927aeba..264ebe1d8e 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -126,6 +126,9 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys) scan->xs_hitup = NULL; scan->xs_hitupdesc = NULL; + /* set in each AM when applicable */ + scan->xs_prefetch = NULL; + return scan; } @@ -440,8 +443,9 @@ systable_beginscan(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* no index prefetch for system catalogs */ sysscan->iscan = index_beginscan(heapRelation, irel, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; } @@ -696,8 +700,9 @@ systable_beginscan_ordered(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* no index prefetch for system catalogs */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index b25b03f7ab..0b8f136f04 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -54,11 +54,13 @@ #include "catalog/pg_amproc.h" #include "catalog/pg_type.h" #include "commands/defrem.h" +#include "common/hashfn.h" #include "nodes/makefuncs.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" #include "storage/predicate.h" +#include "utils/lsyscache.h" #include "utils/ruleutils.h" #include "utils/snapmgr.h" #include "utils/syscache.h" @@ -106,7 +108,10 @@ do { \ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap); + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_target, int prefetch_reset); + +static void index_prefetch(IndexScanDesc scan, ItemPointer tid); /* ---------------------------------------------------------------- @@ -200,18 +205,36 @@ index_insert(Relation indexRelation, * index_beginscan - start a scan of an index with amgettuple * * Caller must be holding suitable locks on the heap and the index. + * + * prefetch_target determines if prefetching is requested for this index scan. + * We need to be able to disable this for two reasons. Firstly, we don't want + * to do prefetching for IOS (where we hope most of the heap pages won't be + * really needed. Secondly, we must prevent infinite loop when determining + * prefetch value for the tablespace - the get_tablespace_io_concurrency() + * does an index scan internally, which would result in infinite loop. So we + * simply disable prefetching in systable_beginscan(). + * + * XXX Maybe we should do prefetching even for catalogs, but then disable it + * when accessing TableSpaceRelationId. We still need the ability to disable + * this and catalogs are expected to be tiny, so prefetching is unlikely to + * make a difference. + * + * XXX The second reason doesn't really apply after effective_io_concurrency + * lookup moved to caller of index_beginscan. */ IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys) + int nkeys, int norderbys, + int prefetch_target, int prefetch_reset) { IndexScanDesc scan; Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false); + scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false, + prefetch_target, prefetch_reset); /* * Save additional parameters into the scandesc. Everything else was set @@ -241,7 +264,8 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false); + scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, + 0, 0); /* no prefetch */ /* * Save additional parameters into the scandesc. Everything else was set @@ -258,7 +282,8 @@ index_beginscan_bitmap(Relation indexRelation, static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap) + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_target, int prefetch_reset) { IndexScanDesc scan; @@ -276,12 +301,27 @@ index_beginscan_internal(Relation indexRelation, /* * Tell the AM to open a scan. */ - scan = indexRelation->rd_indam->ambeginscan(indexRelation, nkeys, - norderbys); + scan = indexRelation->rd_indam->ambeginscan(indexRelation, nkeys, norderbys); /* Initialize information for parallel scan. */ scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; + /* with prefetching enabled, initialize the necessary state */ + if (prefetch_target > 0) + { + IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); + + prefetcher->queueIndex = 0; + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + + prefetcher->prefetchTarget = 0; + prefetcher->prefetchMaxTarget = prefetch_target; + prefetcher->prefetchReset = prefetch_reset; + + scan->xs_prefetch = prefetcher; + } + return scan; } @@ -317,6 +357,20 @@ index_rescan(IndexScanDesc scan, scan->indexRelation->rd_indam->amrescan(scan, keys, nkeys, orderbys, norderbys); + + /* If we're prefetching for this index, maybe reset some of the state. */ + if (scan->xs_prefetch != NULL) + { + IndexPrefetch prefetcher = scan->xs_prefetch; + + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + prefetcher->queueIndex = 0; + prefetcher->prefetchDone = false; + + prefetcher->prefetchTarget = Min(prefetcher->prefetchTarget, + prefetcher->prefetchReset); + } } /* ---------------- @@ -345,6 +399,19 @@ index_endscan(IndexScanDesc scan) if (scan->xs_temp_snap) UnregisterSnapshot(scan->xs_snapshot); + /* If prefetching enabled, log prefetch stats. */ + if (scan->xs_prefetch) + { + IndexPrefetch prefetch = scan->xs_prefetch; + + elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, + prefetch->countAll, + prefetch->countPrefetch, + prefetch->countPrefetch * 100.0 / prefetch->countAll, + prefetch->countSkipCached, + prefetch->countSkipSequential); + } + /* Release the scan data structure itself */ IndexScanEnd(scan); } @@ -487,10 +554,13 @@ index_parallelrescan(IndexScanDesc scan) * index_beginscan_parallel - join parallel index scan * * Caller must be holding suitable locks on the heap and the index. + * + * XXX See index_beginscan() for more comments on prefetch_target. */ IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, - int norderbys, ParallelIndexScanDesc pscan) + int norderbys, ParallelIndexScanDesc pscan, + int prefetch_target, int prefetch_reset) { Snapshot snapshot; IndexScanDesc scan; @@ -499,7 +569,7 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, snapshot = RestoreSnapshot(pscan->ps_snapshot_data); RegisterSnapshot(snapshot); scan = index_beginscan_internal(indexrel, nkeys, norderbys, snapshot, - pscan, true); + pscan, true, prefetch_target, prefetch_reset); /* * Save additional parameters into the scandesc. Everything else was set @@ -623,20 +693,74 @@ index_fetch_heap(IndexScanDesc scan, TupleTableSlot *slot) bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot) { + IndexPrefetch prefetch = scan->xs_prefetch; + for (;;) { + /* with prefetching enabled, accumulate enough TIDs into the prefetch */ + if (PREFETCH_ACTIVE(prefetch)) + { + /* + * incrementally ramp up prefetch distance + * + * XXX Intentionally done as first, so that with prefetching there's + * always at least one item in the queue. + */ + prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, + prefetch->prefetchMaxTarget); + + /* + * get more TID while there is empty space in the queue (considering + * current prefetch target + */ + while (!PREFETCH_FULL(prefetch)) + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* If we're out of index entries, we're done */ + if (tid == NULL) + { + prefetch->prefetchDone = true; + break; + } + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; + prefetch->queueEnd++; + + index_prefetch(scan, tid); + } + } + if (!scan->xs_heap_continue) { - ItemPointer tid; + if (PREFETCH_ENABLED(prefetch)) + { + /* prefetching enabled, but reached the end and queue empty */ + if (PREFETCH_DONE(prefetch)) + break; + + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; + prefetch->queueIndex++; + } + else /* not prefetching, just do the regular work */ + { + ItemPointer tid; - /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); - /* If we're out of index entries, we're done */ - if (tid == NULL) - break; + /* If we're out of index entries, we're done */ + if (tid == NULL) + break; + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + } - Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); } /* @@ -988,3 +1112,258 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, return build_local_reloptions(&relopts, attoptions, validate); } + +/* + * Add the block to the tiny top-level queue (LRU), and check if the block + * is in a sequential pattern. + */ +static bool +index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) +{ + int idx; + + /* If the queue is empty, just store the block and we're done. */ + if (prefetch->blockIndex == 0) + { + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + return false; + } + + /* + * Otherwise, check if it's the same as the immediately preceding block (we + * don't want to prefetch the same block over and over.) + */ + if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) + return true; + + /* Not the same block, so add it to the queue. */ + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + + /* check sequential patter a couple requests back */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) + { + /* not enough requests to confirm a sequential pattern */ + if (prefetch->blockIndex < i) + return false; + + /* + * index of the already requested buffer (-1 because we already + * incremented the index when adding the block to the queue) + */ + idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); + + /* */ + if (prefetch->blockItems[idx] != (block - i)) + return false; + } + + return true; +} + +/* + * index_prefetch_add_cache + * Add a block to the cache, return true if it was recently prefetched. + * + * When checking a block, we need to check if it was recently prefetched, + * where recently means within PREFETCH_CACHE_SIZE requests. This check + * needs to be very cheap, even with fairly large caches (hundreds of + * entries). The cache does not need to be perfect, we can accept false + * positives/negatives, as long as the rate is reasonably low. We also + * need to expire entries, so that only "recent" requests are remembered. + * + * A queue would allow expiring the requests, but checking if a block was + * prefetched would be expensive (linear search for longer queues). Another + * option would be a hash table, but that has issues with expiring entries + * cheaply (which usually degrades the hash table). + * + * So we use a cache that is organized as multiple small LRU caches. Each + * block is mapped to a particular LRU by hashing (so it's a bit like a + * hash table), and each LRU is tiny (e.g. 8 entries). The LRU only keeps + * the most recent requests (for that particular LRU). + * + * This allows quick searches and expiration, with false negatives (when + * a particular LRU has too many collisions). + * + * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 + * prefetch request in total. + * + * The recency is determined using a prefetch counter, incremented every + * time we end up prefetching a block. The counter is uint64, so it should + * not wrap (125 zebibytes, would take ~4 million years at 1GB/s). + * + * To check if a block was prefetched recently, we calculate hash(block), + * and then linearly search if the tiny LRU has entry for the same block + * and request less than PREFETCH_CACHE_SIZE ago. + * + * At the same time, we either update the entry (for the same block) if + * found, or replace the oldest/empty entry. + * + * If the block was not recently prefetched (i.e. we want to prefetch it), + * we increment the counter. + */ +static bool +index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) +{ + PrefetchCacheEntry *entry; + + /* calculate which LRU to use */ + int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; + + /* entry to (maybe) use for this block request */ + uint64 oldestRequest = PG_UINT64_MAX; + int oldestIndex = -1; + + /* + * First add the block to the (tiny) top-level LRU cache and see if it's + * part of a sequential pattern. In this case we just ignore the block + * and don't prefetch it - we expect read-ahead to do a better job. + * + * XXX Maybe we should still add the block to the later cache, in case + * we happen to access it later? That might help if we first scan a lot + * of the table sequentially, and then randomly. Not sure that's very + * likely with index access, though. + */ + if (index_prefetch_is_sequential(prefetch, block)) + { + prefetch->countSkipSequential++; + return true; + } + + /* see if we already have prefetched this block (linear search of LRU) */ + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) + { + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; + + /* Is this the oldest prefetch request in this LRU? */ + if (entry->request < oldestRequest) + { + oldestRequest = entry->request; + oldestIndex = i; + } + + /* Request numbers are positive, so 0 means "unused". */ + if (entry->request == 0) + continue; + + /* Is this entry for the same block as the current request? */ + if (entry->block == block) + { + bool prefetched; + + /* + * Is the old request sufficiently recent? If yes, we treat the + * block as already prefetched. + * + * XXX We do add the cache size to the request in order not to + * have issues with uint64 underflows. + */ + prefetched = (entry->request + PREFETCH_CACHE_SIZE >= prefetch->prefetchReqNumber); + + /* Update the request number. */ + entry->request = ++prefetch->prefetchReqNumber; + + prefetch->countSkipCached += (prefetched) ? 1 : 0; + + return prefetched; + } + } + + /* + * We didn't find the block in the LRU, so store it either in an empty + * entry, or in the "oldest" prefetch request in this LRU. + */ + Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); + + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; + + entry->block = block; + entry->request = ++prefetch->prefetchReqNumber; + + /* not in the prefetch cache */ + return false; +} + +/* + * Do prefetching, and gradually increase the prefetch distance. + * + * XXX This is limited to a single index page (because that's where we get + * currPos.items from). But index tuples are typically very small, so there + * should be quite a bit of stuff to prefetch (especially with deduplicated + * indexes, etc.). Does not seem worth reworking the index access to allow + * more aggressive prefetching, it's best effort. + * + * XXX Some ideas how to auto-tune the prefetching, so that unnecessary + * prefetching does not cause significant regressions (e.g. for nestloop + * with inner index scan). We could track number of index pages visited + * and index tuples returned, to calculate avg tuples / page, and then + * use that to limit prefetching after switching to a new page (instead + * of just using prefetchMaxTarget, which can get much larger). + * + * XXX Obviously, another option is to use the planner estimates - we know + * how many rows we're expected to fetch (on average, assuming the estimates + * are reasonably accurate), so why not to use that. And maybe combine it + * with the auto-tuning based on runtime statistics, described above. + * + * XXX The prefetching may interfere with the patch allowing us to evaluate + * conditions on the index tuple, in which case we may not need the heap + * tuple. Maybe if there's such filter, we should prefetch only pages that + * are not all-visible (and the same idea would also work for IOS), but + * it also makes the indexing a bit "aware" of the visibility stuff (which + * seems a bit wrong). Also, maybe we should consider the filter selectivity + * (if the index-only filter is expected to eliminate only few rows, then + * the vm check is pointless). Maybe this could/should be auto-tuning too, + * i.e. we could track how many heap tuples were needed after all, and then + * we would consider this when deciding whether to prefetch all-visible + * pages or not (matters only for regular index scans, not IOS). + * + * XXX Maybe we could/should also prefetch the next index block, e.g. stored + * in BTScanPosData.nextPage. + */ +static void +index_prefetch(IndexScanDesc scan, ItemPointer tid) +{ + IndexPrefetch prefetch = scan->xs_prefetch; + BlockNumber block; + + /* + * No heap relation means bitmap index scan, which does prefetching at + * the bitmap heap scan, so no prefetch here (we can't do it anyway, + * without the heap) + * + * XXX But in this case we should have prefetchMaxTarget=0, because in + * index_bebinscan_bitmap() we disable prefetching. So maybe we should + * just check that. + */ + if (!prefetch) + return; + + /* was it initialized correctly? */ + // Assert(prefetch->prefetchIndex != -1); + + /* + * If we got here, prefetching is enabled and it's a node that supports + * prefetching (i.e. it can't be a bitmap index scan). + */ + Assert(scan->heapRelation); + + prefetch->countAll++; + + block = ItemPointerGetBlockNumber(tid); + + /* + * Do not prefetch the same block over and over again, + * + * This happens e.g. for clustered or naturally correlated indexes + * (fkey to a sequence ID). It's not expensive (the block is in page + * cache already, so no I/O), but it's not free either. + */ + if (!index_prefetch_add_cache(prefetch, block)) + { + prefetch->countPrefetch++; + + PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); + pgBufferUsage.blks_prefetches++; + } +} diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 8570b14f62..6ae445d62c 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3558,6 +3558,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) !INSTR_TIME_IS_ZERO(usage->blk_write_time)); bool has_temp_timing = (!INSTR_TIME_IS_ZERO(usage->temp_blk_read_time) || !INSTR_TIME_IS_ZERO(usage->temp_blk_write_time)); + bool has_prefetches = (usage->blks_prefetches > 0); bool show_planning = (planning && (has_shared || has_local || has_temp || has_timing || has_temp_timing)); @@ -3655,6 +3656,23 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) appendStringInfoChar(es->str, '\n'); } + /* As above, show only positive counter values. */ + if (has_prefetches) + { + ExplainIndentText(es); + appendStringInfoString(es->str, "Prefetches:"); + + if (usage->blks_prefetches > 0) + appendStringInfo(es->str, " blocks=%lld", + (long long) usage->blks_prefetches); + + if (usage->blks_prefetch_rounds > 0) + appendStringInfo(es->str, " rounds=%lld", + (long long) usage->blks_prefetch_rounds); + + appendStringInfoChar(es->str, '\n'); + } + if (show_planning) es->indent--; } diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 1d82b64b89..e5ce1dbc95 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -765,11 +765,15 @@ check_exclusion_or_unique_constraint(Relation heap, Relation index, /* * May have to restart scan from this point if a potential conflict is * found. + * + * XXX Should this do index prefetch? Probably not worth it for unique + * constraints, I guess? Otherwise we should calculate prefetch_target + * just like in nodeIndexscan etc. */ retry: conflict = false; found_self = false; - index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0); + index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0, 0); index_rescan(index_scan, scankeys, indnkeyatts, NULL, 0); while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot)) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index e776524227..c0bb732658 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -204,8 +204,13 @@ RelationFindReplTupleByIndex(Relation rel, Oid idxoid, /* Build scan key. */ skey_attoff = build_replindex_scan_key(skey, rel, idxrel, searchslot); - /* Start an index scan. */ - scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0); + /* Start an index scan. + * + * XXX Should this do index prefetching? We're looking for a single tuple, + * probably using a PK / UNIQUE index, so does not seem worth it. If we + * reconsider this, calclate prefetch_target like in nodeIndexscan. + */ + scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0, 0); retry: found = false; diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c index ee78a5749d..434be59fca 100644 --- a/src/backend/executor/instrument.c +++ b/src/backend/executor/instrument.c @@ -235,6 +235,8 @@ BufferUsageAdd(BufferUsage *dst, const BufferUsage *add) dst->local_blks_written += add->local_blks_written; dst->temp_blks_read += add->temp_blks_read; dst->temp_blks_written += add->temp_blks_written; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds; + dst->blks_prefetches += add->blks_prefetches; INSTR_TIME_ADD(dst->blk_read_time, add->blk_read_time); INSTR_TIME_ADD(dst->blk_write_time, add->blk_write_time); INSTR_TIME_ADD(dst->temp_blk_read_time, add->temp_blk_read_time); @@ -257,6 +259,8 @@ BufferUsageAccumDiff(BufferUsage *dst, dst->local_blks_written += add->local_blks_written - sub->local_blks_written; dst->temp_blks_read += add->temp_blks_read - sub->temp_blks_read; dst->temp_blks_written += add->temp_blks_written - sub->temp_blks_written; + dst->blks_prefetches += add->blks_prefetches - sub->blks_prefetches; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds - sub->blks_prefetch_rounds; INSTR_TIME_ACCUM_DIFF(dst->blk_read_time, add->blk_read_time, sub->blk_read_time); INSTR_TIME_ACCUM_DIFF(dst->blk_write_time, diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 0b43a9b969..3ecb8470d4 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -87,12 +87,20 @@ IndexOnlyNext(IndexOnlyScanState *node) * We reach here if the index only scan is not parallel, or if we're * serially executing an index only scan that was planned to be * parallel. + * + * XXX Maybe we should enable prefetching, but prefetch only pages that + * are not all-visible (but checking that from the index code seems like + * a violation of layering etc). + * + * XXX This might lead to IOS being slower than plain index scan, if the + * table has a lot of pages that need recheck. */ scandesc = index_beginscan(node->ss.ss_currentRelation, node->ioss_RelationDesc, estate->es_snapshot, node->ioss_NumScanKeys, - node->ioss_NumOrderByKeys); + node->ioss_NumOrderByKeys, + 0, 0); /* no index prefetch for IOS */ node->ioss_ScanDesc = scandesc; @@ -674,7 +682,8 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + 0, 0); /* no index prefetch for IOS */ node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -719,7 +728,8 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + 0, 0); /* no index prefetch for IOS */ node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 4540c7781d..71ae6a47ce 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -43,6 +43,7 @@ #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "utils/spccache.h" /* * When an ordering operator is used, tuples fetched from the index that @@ -85,6 +86,7 @@ IndexNext(IndexScanState *node) ScanDirection direction; IndexScanDesc scandesc; TupleTableSlot *slot; + Relation heapRel = node->ss.ss_currentRelation; /* * extract necessary information from index scan node @@ -103,6 +105,22 @@ IndexNext(IndexScanState *node) if (scandesc == NULL) { + int prefetch_target; + int prefetch_reset; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -111,7 +129,9 @@ IndexNext(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_target, + prefetch_reset); node->iss_ScanDesc = scandesc; @@ -198,6 +218,23 @@ IndexNextWithReorder(IndexScanState *node) if (scandesc == NULL) { + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_target; + int prefetch_reset; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -206,7 +243,9 @@ IndexNextWithReorder(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_target, + prefetch_reset); node->iss_ScanDesc = scandesc; @@ -1678,6 +1717,21 @@ ExecIndexScanInitializeDSM(IndexScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel; + int prefetch_target; + int prefetch_reset; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Maybe reduce the value with parallel workers? + */ + heapRel = node->ss.ss_currentRelation; + + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -1690,7 +1744,9 @@ ExecIndexScanInitializeDSM(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_target, + prefetch_reset); /* * If no run-time keys to calculate or they are ready, go ahead and pass @@ -1726,6 +1782,14 @@ ExecIndexScanInitializeWorker(IndexScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel; + int prefetch_target; + int prefetch_reset; + + heapRel = node->ss.ss_currentRelation; + + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->iss_ScanDesc = @@ -1733,7 +1797,9 @@ ExecIndexScanInitializeWorker(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_target, + prefetch_reset); /* * If no run-time keys to calculate or they are ready, go ahead and pass diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d27ef2985d..d65575fd10 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1131,6 +1131,8 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) need_full_snapshot = true; } + elog(LOG, "slot = %s need_full_snapshot = %d", cmd->slotname, need_full_snapshot); + ctx = CreateInitDecodingContext(cmd->plugin, NIL, need_full_snapshot, InvalidXLogRecPtr, XL_ROUTINE(.page_read = logical_read_xlog_page, diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index c4fcd0076e..0b02b6265d 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -6218,7 +6218,7 @@ get_actual_variable_endpoint(Relation heapRel, index_scan = index_beginscan(heapRel, indexRel, &SnapshotNonVacuumable, - 1, 0); + 1, 0, 0, 0); /* XXX maybe do prefetch? */ /* Set it up for index-only scan */ index_scan->xs_want_itup = true; index_rescan(index_scan, scankeys, 1, NULL, 0); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index a308795665..f3efffc4a8 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -17,6 +17,7 @@ #include "access/sdir.h" #include "access/skey.h" #include "nodes/tidbitmap.h" +#include "storage/bufmgr.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -152,7 +153,9 @@ extern bool index_insert(Relation indexRelation, extern IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys); + int nkeys, int norderbys, + int prefetch_target, + int prefetch_reset); extern IndexScanDesc index_beginscan_bitmap(Relation indexRelation, Snapshot snapshot, int nkeys); @@ -169,7 +172,9 @@ extern void index_parallelscan_initialize(Relation heapRelation, extern void index_parallelrescan(IndexScanDesc scan); extern IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, - ParallelIndexScanDesc pscan); + ParallelIndexScanDesc pscan, + int prefetch_target, + int prefetch_reset); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); struct TupleTableSlot; @@ -230,4 +235,108 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); +/* + * XXX not sure it's the right place to define these callbacks etc. + */ +typedef void (*prefetcher_getrange_function) (IndexScanDesc scandesc, + ScanDirection direction, + int *start, int *end, + bool *reset); + +typedef BlockNumber (*prefetcher_getblock_function) (IndexScanDesc scandesc, + ScanDirection direction, + int index); + +/* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. Doesn't need to be perfectly accurate, but we + * aim to make false positives/negatives reasonably low. + */ +typedef struct PrefetchCacheEntry { + BlockNumber block; + uint64 request; +} PrefetchCacheEntry; + +/* + * Size of the cache of recently prefetched blocks - shouldn't be too + * small or too large. 1024 seems about right, it covers ~8MB of data. + * It's somewhat arbitrary, there's no particular formula saying it + * should not be higher/lower. + * + * The cache is structured as an array of small LRU caches, so the total + * size needs to be a multiple of LRU size. The LRU should be tiny to + * keep linear search cheap enough. + * + * XXX Maybe we could consider effective_cache_size or something? + */ +#define PREFETCH_LRU_SIZE 8 +#define PREFETCH_LRU_COUNT 128 +#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) + +/* + * Used to detect sequential patterns (and disable prefetching). + */ +#define PREFETCH_QUEUE_HISTORY 8 +#define PREFETCH_SEQ_PATTERN_BLOCKS 4 + + +typedef struct IndexPrefetchData +{ + /* + * XXX We need to disable this in some cases (e.g. when using index-only + * scans, we don't want to prefetch pages). Or maybe we should prefetch + * only pages that are not all-visible, that'd be even better. + */ + int prefetchTarget; /* how far we should be prefetching */ + int prefetchMaxTarget; /* maximum prefetching distance */ + int prefetchReset; /* reset to this distance on rescan */ + bool prefetchDone; /* did we get all TIDs from the index? */ + + /* runtime statistics */ + uint64 countAll; /* all prefetch requests */ + uint64 countPrefetch; /* actual prefetches */ + uint64 countSkipSequential; + uint64 countSkipCached; + + /* + * Queue of TIDs to prefetch. + * + * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler + * than dynamically adjusting for custom values. + */ + ItemPointerData queueItems[MAX_IO_CONCURRENCY]; + uint64 queueIndex; /* next TID to prefetch */ + uint64 queueStart; /* first valid TID in queue */ + uint64 queueEnd; /* first invalid (empty) TID in queue */ + + /* + * A couple of last prefetched blocks, used to check for certain access + * pattern and skip prefetching - e.g. for sequential access). + * + * XXX Separate from the main queue, because we only want to compare the + * block numbers, not the whole TID. In sequential access it's likely we + * read many items from each page, and we don't want to check many items + * (as that is much more expensive). + */ + BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; + uint64 blockIndex; /* index in the block (points to the first + * empty entry)*/ + + /* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. + */ + uint64 prefetchReqNumber; + PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; + +} IndexPrefetchData; + +#define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) +#define PREFETCH_QUEUE_EMPTY(p) ((p)->queueEnd == (p)->queueIndex) +#define PREFETCH_ENABLED(p) ((p) && ((p)->prefetchMaxTarget > 0)) +#define PREFETCH_FULL(p) ((p)->queueEnd - (p)->queueIndex == (p)->prefetchTarget) +#define PREFETCH_DONE(p) ((p) && ((p)->prefetchDone && PREFETCH_QUEUE_EMPTY(p))) +#define PREFETCH_ACTIVE(p) (PREFETCH_ENABLED(p) && !(p)->prefetchDone) +#define PREFETCH_BLOCK_INDEX(v) ((v) % PREFETCH_QUEUE_HISTORY) + #endif /* GENAM_H */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index d03360eac0..c119fe597d 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -106,6 +106,12 @@ typedef struct IndexFetchTableData Relation rel; } IndexFetchTableData; +/* + * Forward declaration, defined in genam.h. + */ +typedef struct IndexPrefetchData IndexPrefetchData; +typedef struct IndexPrefetchData *IndexPrefetch; + /* * We use the same IndexScanDescData structure for both amgettuple-based * and amgetbitmap-based index scans. Some fields are only relevant in @@ -162,6 +168,9 @@ typedef struct IndexScanDescData bool *xs_orderbynulls; bool xs_recheckorderby; + /* prefetching state (or NULL if disabled) */ + IndexPrefetchData *xs_prefetch; + /* parallel index scan information, in shared memory */ struct ParallelIndexScanDescData *parallel_scan; } IndexScanDescData; diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index 87e5e2183b..97dd3c2c42 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -33,6 +33,8 @@ typedef struct BufferUsage int64 local_blks_written; /* # of local disk blocks written */ int64 temp_blks_read; /* # of temp blocks read */ int64 temp_blks_written; /* # of temp blocks written */ + int64 blks_prefetch_rounds; /* # of prefetch rounds */ + int64 blks_prefetches; /* # of buffers prefetched */ instr_time blk_read_time; /* time spent reading blocks */ instr_time blk_write_time; /* time spent writing blocks */ instr_time temp_blk_read_time; /* time spent reading temp blocks */ ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-10-16 15:34 Tomas Vondra <[email protected]> parent: Tomas Vondra <[email protected]> 1 sibling, 1 reply; 10+ messages in thread From: Tomas Vondra @ 2023-10-16 15:34 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> Hi, Attached is a v6 of the patch, which rebases v5 (just some minor bitrot), and also does a couple changes which I kept in separate patches to make it obvious what changed. 0001-v5-20231016.patch ---------------------- Rebase to current master. 0002-comments-and-minor-cleanup-20231012.patch ---------------------------------------------- Various comment improvements (remove obsolete ones clarify a bunch of other comments, etc.). I tried to explain the reasoning why some places disable prefetching (e.g. in catalogs, replication, ...), explain how the caching / LRU works etc. 0003-remove-prefetch_reset-20231016.patch ----------------------------------------- I decided to remove the separate prefetch_reset parameter, so that all the index_beginscan() methods only take a parameter specifying the maximum prefetch target. The reset was added early when the prefetch happened much lower in the AM code, at the index page level, and the reset was when moving to the next index page. But now after the prefetch moved to the executor, this doesn't make much sense - the resets happen on rescans, and it seems right to just reset to 0 (just like for bitmap heap scans). 0004-PoC-prefetch-for-IOS-20231016.patch ---------------------------------------- This is a PoC adding the prefetch to index-only scans too. At first that may seem rather strange, considering eliminating the heap fetches is the whole point of IOS. But if the pages are not marked as all-visible (say, the most recent part of the table), we may still have to fetch them. In which case it'd be easy to see cases that IOS is slower than a regular index scan (with prefetching). The code is quite rough. It adds a separate index_getnext_tid_prefetch() function, adding prefetching on top of index_getnext_tid(). I'm not sure it's the right pattern, but it's pretty much what index_getnext_slot() does too, except that it also does the fetch + store to the slot. Note: There's a second patch adding index-only filters, which requires the regular index scans from index_getnext_slot() to _tid() too. The prefetching then happens only after checking the visibility map (if requested). This part definitely needs improvements - for example there's no attempt to reuse the VM buffer, which I guess might be expensive. index-prefetch.pdf ------------------ Attached is also a PDF with results of the same benchmark I did before, comparing master vs. patched with various data patterns and scan types. It's not 100% comparable to earlier results as I only ran it on a laptop, and it's a bit noisier too. The overall behavior and conclusions are however the same. I was specifically interested in the IOS behavior, so I added two more cases to test - indexonlyscan and indexonlyscan-clean. The first is the worst-case scenario, with no pages marked as all-visible in VM (the test simply deletes the VM), while indexonlyscan-clean is the good-case (no heap fetches needed). The results mostly match the expected behavior, particularly for the uncached runs (when the data is expected to not be in memory): * indexonlyscan (i.e. bad case) - About the same results as "indexscans", with the same speedups etc. Which is a good thing (i.e. IOS is not unexpectedly slower than regular indexscans). * indexonlyscan-clean (i.e. good case) - Seems to have mostly the same performance as without the prefetching, except for the low-cardinality runs with many rows per key. I haven't checked what's causing this, but I'd bet it's the extra buffer lookups/management I mentioned. I noticed there's another prefetching-related patch [1] from Thomas Munro. I haven't looked at it yet, so hard to say how much it interferes with this patch. But the idea looks interesting. [1] https://www.postgresql.org/message-id/flat/CA+hUKGJkOiOCa+mag4BF+zHo7qo=o9CFheB8=g6uT5TUm2gkvA@mail.... regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company Attachments: [application/pdf] index-prefetch.pdf (109.4K, 2-index-prefetch.pdf) download [text/x-patch] 0001-v5-20231012.patch (40.8K, 3-0001-v5-20231012.patch) download | inline diff: From 2faea9a5ef9b584853341489c9e30f11129638c0 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Fri, 13 Oct 2023 22:33:51 +0200 Subject: [PATCH 1/4] v5 --- src/backend/access/gist/gistget.c | 1 - src/backend/access/heap/heapam_handler.c | 7 +- src/backend/access/index/genam.c | 9 +- src/backend/access/index/indexam.c | 411 ++++++++++++++++++++++- src/backend/commands/explain.c | 18 + src/backend/executor/execIndexing.c | 6 +- src/backend/executor/execReplication.c | 9 +- src/backend/executor/instrument.c | 4 + src/backend/executor/nodeIndexonlyscan.c | 16 +- src/backend/executor/nodeIndexscan.c | 74 +++- src/backend/replication/walsender.c | 2 + src/backend/utils/adt/selfuncs.c | 2 +- src/include/access/genam.h | 113 ++++++- src/include/access/relscan.h | 9 + src/include/executor/instrument.h | 2 + 15 files changed, 650 insertions(+), 33 deletions(-) diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c index 31349174280..3acfa762e7f 100644 --- a/src/backend/access/gist/gistget.c +++ b/src/backend/access/gist/gistget.c @@ -677,7 +677,6 @@ gistgettuple(IndexScanDesc scan, ScanDirection dir) scan->xs_hitup = so->pageData[so->curPageData].recontup; so->curPageData++; - return true; } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 7c28dafb728..46c85751cf2 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -44,6 +44,7 @@ #include "storage/smgr.h" #include "utils/builtins.h" #include "utils/rel.h" +#include "utils/spccache.h" static void reform_and_rewrite_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, @@ -747,6 +748,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, PROGRESS_CLUSTER_INDEX_RELID }; int64 ci_val[2]; + int prefetch_target; + + prefetch_target = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; @@ -755,7 +759,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tableScan = NULL; heapScan = NULL; - indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0); + indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0, + prefetch_target, prefetch_target); index_rescan(indexScan, NULL, 0, NULL, 0); } else diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 4ca12006843..230667f888b 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -126,6 +126,9 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys) scan->xs_hitup = NULL; scan->xs_hitupdesc = NULL; + /* set in each AM when applicable */ + scan->xs_prefetch = NULL; + return scan; } @@ -440,8 +443,9 @@ systable_beginscan(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* no index prefetch for system catalogs */ sysscan->iscan = index_beginscan(heapRelation, irel, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; } @@ -696,8 +700,9 @@ systable_beginscan_ordered(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* no index prefetch for system catalogs */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index b25b03f7abc..0b8f136f042 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -54,11 +54,13 @@ #include "catalog/pg_amproc.h" #include "catalog/pg_type.h" #include "commands/defrem.h" +#include "common/hashfn.h" #include "nodes/makefuncs.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" #include "storage/predicate.h" +#include "utils/lsyscache.h" #include "utils/ruleutils.h" #include "utils/snapmgr.h" #include "utils/syscache.h" @@ -106,7 +108,10 @@ do { \ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap); + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_target, int prefetch_reset); + +static void index_prefetch(IndexScanDesc scan, ItemPointer tid); /* ---------------------------------------------------------------- @@ -200,18 +205,36 @@ index_insert(Relation indexRelation, * index_beginscan - start a scan of an index with amgettuple * * Caller must be holding suitable locks on the heap and the index. + * + * prefetch_target determines if prefetching is requested for this index scan. + * We need to be able to disable this for two reasons. Firstly, we don't want + * to do prefetching for IOS (where we hope most of the heap pages won't be + * really needed. Secondly, we must prevent infinite loop when determining + * prefetch value for the tablespace - the get_tablespace_io_concurrency() + * does an index scan internally, which would result in infinite loop. So we + * simply disable prefetching in systable_beginscan(). + * + * XXX Maybe we should do prefetching even for catalogs, but then disable it + * when accessing TableSpaceRelationId. We still need the ability to disable + * this and catalogs are expected to be tiny, so prefetching is unlikely to + * make a difference. + * + * XXX The second reason doesn't really apply after effective_io_concurrency + * lookup moved to caller of index_beginscan. */ IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys) + int nkeys, int norderbys, + int prefetch_target, int prefetch_reset) { IndexScanDesc scan; Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false); + scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false, + prefetch_target, prefetch_reset); /* * Save additional parameters into the scandesc. Everything else was set @@ -241,7 +264,8 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false); + scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, + 0, 0); /* no prefetch */ /* * Save additional parameters into the scandesc. Everything else was set @@ -258,7 +282,8 @@ index_beginscan_bitmap(Relation indexRelation, static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap) + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_target, int prefetch_reset) { IndexScanDesc scan; @@ -276,12 +301,27 @@ index_beginscan_internal(Relation indexRelation, /* * Tell the AM to open a scan. */ - scan = indexRelation->rd_indam->ambeginscan(indexRelation, nkeys, - norderbys); + scan = indexRelation->rd_indam->ambeginscan(indexRelation, nkeys, norderbys); /* Initialize information for parallel scan. */ scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; + /* with prefetching enabled, initialize the necessary state */ + if (prefetch_target > 0) + { + IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); + + prefetcher->queueIndex = 0; + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + + prefetcher->prefetchTarget = 0; + prefetcher->prefetchMaxTarget = prefetch_target; + prefetcher->prefetchReset = prefetch_reset; + + scan->xs_prefetch = prefetcher; + } + return scan; } @@ -317,6 +357,20 @@ index_rescan(IndexScanDesc scan, scan->indexRelation->rd_indam->amrescan(scan, keys, nkeys, orderbys, norderbys); + + /* If we're prefetching for this index, maybe reset some of the state. */ + if (scan->xs_prefetch != NULL) + { + IndexPrefetch prefetcher = scan->xs_prefetch; + + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + prefetcher->queueIndex = 0; + prefetcher->prefetchDone = false; + + prefetcher->prefetchTarget = Min(prefetcher->prefetchTarget, + prefetcher->prefetchReset); + } } /* ---------------- @@ -345,6 +399,19 @@ index_endscan(IndexScanDesc scan) if (scan->xs_temp_snap) UnregisterSnapshot(scan->xs_snapshot); + /* If prefetching enabled, log prefetch stats. */ + if (scan->xs_prefetch) + { + IndexPrefetch prefetch = scan->xs_prefetch; + + elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, + prefetch->countAll, + prefetch->countPrefetch, + prefetch->countPrefetch * 100.0 / prefetch->countAll, + prefetch->countSkipCached, + prefetch->countSkipSequential); + } + /* Release the scan data structure itself */ IndexScanEnd(scan); } @@ -487,10 +554,13 @@ index_parallelrescan(IndexScanDesc scan) * index_beginscan_parallel - join parallel index scan * * Caller must be holding suitable locks on the heap and the index. + * + * XXX See index_beginscan() for more comments on prefetch_target. */ IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, - int norderbys, ParallelIndexScanDesc pscan) + int norderbys, ParallelIndexScanDesc pscan, + int prefetch_target, int prefetch_reset) { Snapshot snapshot; IndexScanDesc scan; @@ -499,7 +569,7 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, snapshot = RestoreSnapshot(pscan->ps_snapshot_data); RegisterSnapshot(snapshot); scan = index_beginscan_internal(indexrel, nkeys, norderbys, snapshot, - pscan, true); + pscan, true, prefetch_target, prefetch_reset); /* * Save additional parameters into the scandesc. Everything else was set @@ -623,20 +693,74 @@ index_fetch_heap(IndexScanDesc scan, TupleTableSlot *slot) bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot) { + IndexPrefetch prefetch = scan->xs_prefetch; + for (;;) { + /* with prefetching enabled, accumulate enough TIDs into the prefetch */ + if (PREFETCH_ACTIVE(prefetch)) + { + /* + * incrementally ramp up prefetch distance + * + * XXX Intentionally done as first, so that with prefetching there's + * always at least one item in the queue. + */ + prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, + prefetch->prefetchMaxTarget); + + /* + * get more TID while there is empty space in the queue (considering + * current prefetch target + */ + while (!PREFETCH_FULL(prefetch)) + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* If we're out of index entries, we're done */ + if (tid == NULL) + { + prefetch->prefetchDone = true; + break; + } + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; + prefetch->queueEnd++; + + index_prefetch(scan, tid); + } + } + if (!scan->xs_heap_continue) { - ItemPointer tid; + if (PREFETCH_ENABLED(prefetch)) + { + /* prefetching enabled, but reached the end and queue empty */ + if (PREFETCH_DONE(prefetch)) + break; + + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; + prefetch->queueIndex++; + } + else /* not prefetching, just do the regular work */ + { + ItemPointer tid; - /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); - /* If we're out of index entries, we're done */ - if (tid == NULL) - break; + /* If we're out of index entries, we're done */ + if (tid == NULL) + break; + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + } - Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); } /* @@ -988,3 +1112,258 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, return build_local_reloptions(&relopts, attoptions, validate); } + +/* + * Add the block to the tiny top-level queue (LRU), and check if the block + * is in a sequential pattern. + */ +static bool +index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) +{ + int idx; + + /* If the queue is empty, just store the block and we're done. */ + if (prefetch->blockIndex == 0) + { + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + return false; + } + + /* + * Otherwise, check if it's the same as the immediately preceding block (we + * don't want to prefetch the same block over and over.) + */ + if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) + return true; + + /* Not the same block, so add it to the queue. */ + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + + /* check sequential patter a couple requests back */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) + { + /* not enough requests to confirm a sequential pattern */ + if (prefetch->blockIndex < i) + return false; + + /* + * index of the already requested buffer (-1 because we already + * incremented the index when adding the block to the queue) + */ + idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); + + /* */ + if (prefetch->blockItems[idx] != (block - i)) + return false; + } + + return true; +} + +/* + * index_prefetch_add_cache + * Add a block to the cache, return true if it was recently prefetched. + * + * When checking a block, we need to check if it was recently prefetched, + * where recently means within PREFETCH_CACHE_SIZE requests. This check + * needs to be very cheap, even with fairly large caches (hundreds of + * entries). The cache does not need to be perfect, we can accept false + * positives/negatives, as long as the rate is reasonably low. We also + * need to expire entries, so that only "recent" requests are remembered. + * + * A queue would allow expiring the requests, but checking if a block was + * prefetched would be expensive (linear search for longer queues). Another + * option would be a hash table, but that has issues with expiring entries + * cheaply (which usually degrades the hash table). + * + * So we use a cache that is organized as multiple small LRU caches. Each + * block is mapped to a particular LRU by hashing (so it's a bit like a + * hash table), and each LRU is tiny (e.g. 8 entries). The LRU only keeps + * the most recent requests (for that particular LRU). + * + * This allows quick searches and expiration, with false negatives (when + * a particular LRU has too many collisions). + * + * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 + * prefetch request in total. + * + * The recency is determined using a prefetch counter, incremented every + * time we end up prefetching a block. The counter is uint64, so it should + * not wrap (125 zebibytes, would take ~4 million years at 1GB/s). + * + * To check if a block was prefetched recently, we calculate hash(block), + * and then linearly search if the tiny LRU has entry for the same block + * and request less than PREFETCH_CACHE_SIZE ago. + * + * At the same time, we either update the entry (for the same block) if + * found, or replace the oldest/empty entry. + * + * If the block was not recently prefetched (i.e. we want to prefetch it), + * we increment the counter. + */ +static bool +index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) +{ + PrefetchCacheEntry *entry; + + /* calculate which LRU to use */ + int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; + + /* entry to (maybe) use for this block request */ + uint64 oldestRequest = PG_UINT64_MAX; + int oldestIndex = -1; + + /* + * First add the block to the (tiny) top-level LRU cache and see if it's + * part of a sequential pattern. In this case we just ignore the block + * and don't prefetch it - we expect read-ahead to do a better job. + * + * XXX Maybe we should still add the block to the later cache, in case + * we happen to access it later? That might help if we first scan a lot + * of the table sequentially, and then randomly. Not sure that's very + * likely with index access, though. + */ + if (index_prefetch_is_sequential(prefetch, block)) + { + prefetch->countSkipSequential++; + return true; + } + + /* see if we already have prefetched this block (linear search of LRU) */ + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) + { + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; + + /* Is this the oldest prefetch request in this LRU? */ + if (entry->request < oldestRequest) + { + oldestRequest = entry->request; + oldestIndex = i; + } + + /* Request numbers are positive, so 0 means "unused". */ + if (entry->request == 0) + continue; + + /* Is this entry for the same block as the current request? */ + if (entry->block == block) + { + bool prefetched; + + /* + * Is the old request sufficiently recent? If yes, we treat the + * block as already prefetched. + * + * XXX We do add the cache size to the request in order not to + * have issues with uint64 underflows. + */ + prefetched = (entry->request + PREFETCH_CACHE_SIZE >= prefetch->prefetchReqNumber); + + /* Update the request number. */ + entry->request = ++prefetch->prefetchReqNumber; + + prefetch->countSkipCached += (prefetched) ? 1 : 0; + + return prefetched; + } + } + + /* + * We didn't find the block in the LRU, so store it either in an empty + * entry, or in the "oldest" prefetch request in this LRU. + */ + Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); + + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; + + entry->block = block; + entry->request = ++prefetch->prefetchReqNumber; + + /* not in the prefetch cache */ + return false; +} + +/* + * Do prefetching, and gradually increase the prefetch distance. + * + * XXX This is limited to a single index page (because that's where we get + * currPos.items from). But index tuples are typically very small, so there + * should be quite a bit of stuff to prefetch (especially with deduplicated + * indexes, etc.). Does not seem worth reworking the index access to allow + * more aggressive prefetching, it's best effort. + * + * XXX Some ideas how to auto-tune the prefetching, so that unnecessary + * prefetching does not cause significant regressions (e.g. for nestloop + * with inner index scan). We could track number of index pages visited + * and index tuples returned, to calculate avg tuples / page, and then + * use that to limit prefetching after switching to a new page (instead + * of just using prefetchMaxTarget, which can get much larger). + * + * XXX Obviously, another option is to use the planner estimates - we know + * how many rows we're expected to fetch (on average, assuming the estimates + * are reasonably accurate), so why not to use that. And maybe combine it + * with the auto-tuning based on runtime statistics, described above. + * + * XXX The prefetching may interfere with the patch allowing us to evaluate + * conditions on the index tuple, in which case we may not need the heap + * tuple. Maybe if there's such filter, we should prefetch only pages that + * are not all-visible (and the same idea would also work for IOS), but + * it also makes the indexing a bit "aware" of the visibility stuff (which + * seems a bit wrong). Also, maybe we should consider the filter selectivity + * (if the index-only filter is expected to eliminate only few rows, then + * the vm check is pointless). Maybe this could/should be auto-tuning too, + * i.e. we could track how many heap tuples were needed after all, and then + * we would consider this when deciding whether to prefetch all-visible + * pages or not (matters only for regular index scans, not IOS). + * + * XXX Maybe we could/should also prefetch the next index block, e.g. stored + * in BTScanPosData.nextPage. + */ +static void +index_prefetch(IndexScanDesc scan, ItemPointer tid) +{ + IndexPrefetch prefetch = scan->xs_prefetch; + BlockNumber block; + + /* + * No heap relation means bitmap index scan, which does prefetching at + * the bitmap heap scan, so no prefetch here (we can't do it anyway, + * without the heap) + * + * XXX But in this case we should have prefetchMaxTarget=0, because in + * index_bebinscan_bitmap() we disable prefetching. So maybe we should + * just check that. + */ + if (!prefetch) + return; + + /* was it initialized correctly? */ + // Assert(prefetch->prefetchIndex != -1); + + /* + * If we got here, prefetching is enabled and it's a node that supports + * prefetching (i.e. it can't be a bitmap index scan). + */ + Assert(scan->heapRelation); + + prefetch->countAll++; + + block = ItemPointerGetBlockNumber(tid); + + /* + * Do not prefetch the same block over and over again, + * + * This happens e.g. for clustered or naturally correlated indexes + * (fkey to a sequence ID). It's not expensive (the block is in page + * cache already, so no I/O), but it's not free either. + */ + if (!index_prefetch_add_cache(prefetch, block)) + { + prefetch->countPrefetch++; + + PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); + pgBufferUsage.blks_prefetches++; + } +} diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 13217807eed..8837da91857 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3566,6 +3566,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) !INSTR_TIME_IS_ZERO(usage->blk_write_time)); bool has_temp_timing = (!INSTR_TIME_IS_ZERO(usage->temp_blk_read_time) || !INSTR_TIME_IS_ZERO(usage->temp_blk_write_time)); + bool has_prefetches = (usage->blks_prefetches > 0); bool show_planning = (planning && (has_shared || has_local || has_temp || has_timing || has_temp_timing)); @@ -3663,6 +3664,23 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) appendStringInfoChar(es->str, '\n'); } + /* As above, show only positive counter values. */ + if (has_prefetches) + { + ExplainIndentText(es); + appendStringInfoString(es->str, "Prefetches:"); + + if (usage->blks_prefetches > 0) + appendStringInfo(es->str, " blocks=%lld", + (long long) usage->blks_prefetches); + + if (usage->blks_prefetch_rounds > 0) + appendStringInfo(es->str, " rounds=%lld", + (long long) usage->blks_prefetch_rounds); + + appendStringInfoChar(es->str, '\n'); + } + if (show_planning) es->indent--; } diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 3c6730632de..09418f715fa 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -765,11 +765,15 @@ check_exclusion_or_unique_constraint(Relation heap, Relation index, /* * May have to restart scan from this point if a potential conflict is * found. + * + * XXX Should this do index prefetch? Probably not worth it for unique + * constraints, I guess? Otherwise we should calculate prefetch_target + * just like in nodeIndexscan etc. */ retry: conflict = false; found_self = false; - index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0); + index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0, 0); index_rescan(index_scan, scankeys, indnkeyatts, NULL, 0); while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot)) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 81f27042bc4..f3e1a8d22a4 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -204,8 +204,13 @@ RelationFindReplTupleByIndex(Relation rel, Oid idxoid, /* Build scan key. */ skey_attoff = build_replindex_scan_key(skey, rel, idxrel, searchslot); - /* Start an index scan. */ - scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0); + /* Start an index scan. + * + * XXX Should this do index prefetching? We're looking for a single tuple, + * probably using a PK / UNIQUE index, so does not seem worth it. If we + * reconsider this, calclate prefetch_target like in nodeIndexscan. + */ + scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0, 0); retry: found = false; diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c index ee78a5749d2..434be59fca0 100644 --- a/src/backend/executor/instrument.c +++ b/src/backend/executor/instrument.c @@ -235,6 +235,8 @@ BufferUsageAdd(BufferUsage *dst, const BufferUsage *add) dst->local_blks_written += add->local_blks_written; dst->temp_blks_read += add->temp_blks_read; dst->temp_blks_written += add->temp_blks_written; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds; + dst->blks_prefetches += add->blks_prefetches; INSTR_TIME_ADD(dst->blk_read_time, add->blk_read_time); INSTR_TIME_ADD(dst->blk_write_time, add->blk_write_time); INSTR_TIME_ADD(dst->temp_blk_read_time, add->temp_blk_read_time); @@ -257,6 +259,8 @@ BufferUsageAccumDiff(BufferUsage *dst, dst->local_blks_written += add->local_blks_written - sub->local_blks_written; dst->temp_blks_read += add->temp_blks_read - sub->temp_blks_read; dst->temp_blks_written += add->temp_blks_written - sub->temp_blks_written; + dst->blks_prefetches += add->blks_prefetches - sub->blks_prefetches; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds - sub->blks_prefetch_rounds; INSTR_TIME_ACCUM_DIFF(dst->blk_read_time, add->blk_read_time, sub->blk_read_time); INSTR_TIME_ACCUM_DIFF(dst->blk_write_time, diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index f1db35665c8..75b44db33c6 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -87,12 +87,20 @@ IndexOnlyNext(IndexOnlyScanState *node) * We reach here if the index only scan is not parallel, or if we're * serially executing an index only scan that was planned to be * parallel. + * + * XXX Maybe we should enable prefetching, but prefetch only pages that + * are not all-visible (but checking that from the index code seems like + * a violation of layering etc). + * + * XXX This might lead to IOS being slower than plain index scan, if the + * table has a lot of pages that need recheck. */ scandesc = index_beginscan(node->ss.ss_currentRelation, node->ioss_RelationDesc, estate->es_snapshot, node->ioss_NumScanKeys, - node->ioss_NumOrderByKeys); + node->ioss_NumOrderByKeys, + 0, 0); /* no index prefetch for IOS */ node->ioss_ScanDesc = scandesc; @@ -658,7 +666,8 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + 0, 0); /* no index prefetch for IOS */ node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -703,7 +712,8 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + 0, 0); /* no index prefetch for IOS */ node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 14b9c00217a..185ff0f1449 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -43,6 +43,7 @@ #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "utils/spccache.h" /* * When an ordering operator is used, tuples fetched from the index that @@ -85,6 +86,7 @@ IndexNext(IndexScanState *node) ScanDirection direction; IndexScanDesc scandesc; TupleTableSlot *slot; + Relation heapRel = node->ss.ss_currentRelation; /* * extract necessary information from index scan node @@ -103,6 +105,22 @@ IndexNext(IndexScanState *node) if (scandesc == NULL) { + int prefetch_target; + int prefetch_reset; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -111,7 +129,9 @@ IndexNext(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_target, + prefetch_reset); node->iss_ScanDesc = scandesc; @@ -198,6 +218,23 @@ IndexNextWithReorder(IndexScanState *node) if (scandesc == NULL) { + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_target; + int prefetch_reset; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -206,7 +243,9 @@ IndexNextWithReorder(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_target, + prefetch_reset); node->iss_ScanDesc = scandesc; @@ -1662,6 +1701,21 @@ ExecIndexScanInitializeDSM(IndexScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel; + int prefetch_target; + int prefetch_reset; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Maybe reduce the value with parallel workers? + */ + heapRel = node->ss.ss_currentRelation; + + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -1674,7 +1728,9 @@ ExecIndexScanInitializeDSM(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_target, + prefetch_reset); /* * If no run-time keys to calculate or they are ready, go ahead and pass @@ -1710,6 +1766,14 @@ ExecIndexScanInitializeWorker(IndexScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel; + int prefetch_target; + int prefetch_reset; + + heapRel = node->ss.ss_currentRelation; + + prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); + prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->iss_ScanDesc = @@ -1717,7 +1781,9 @@ ExecIndexScanInitializeWorker(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_target, + prefetch_reset); /* * If no run-time keys to calculate or they are ready, go ahead and pass diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index e250b0567eb..47093cc9cf1 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1131,6 +1131,8 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) need_full_snapshot = true; } + elog(LOG, "slot = %s need_full_snapshot = %d", cmd->slotname, need_full_snapshot); + ctx = CreateInitDecodingContext(cmd->plugin, NIL, need_full_snapshot, InvalidXLogRecPtr, XL_ROUTINE(.page_read = logical_read_xlog_page, diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index c4fcd0076ea..0b02b6265d0 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -6218,7 +6218,7 @@ get_actual_variable_endpoint(Relation heapRel, index_scan = index_beginscan(heapRel, indexRel, &SnapshotNonVacuumable, - 1, 0); + 1, 0, 0, 0); /* XXX maybe do prefetch? */ /* Set it up for index-only scan */ index_scan->xs_want_itup = true; index_rescan(index_scan, scankeys, 1, NULL, 0); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 4e626c615e7..b814af4b2f6 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -17,6 +17,7 @@ #include "access/sdir.h" #include "access/skey.h" #include "nodes/tidbitmap.h" +#include "storage/bufmgr.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -152,7 +153,9 @@ extern bool index_insert(Relation indexRelation, extern IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys); + int nkeys, int norderbys, + int prefetch_target, + int prefetch_reset); extern IndexScanDesc index_beginscan_bitmap(Relation indexRelation, Snapshot snapshot, int nkeys); @@ -169,7 +172,9 @@ extern void index_parallelscan_initialize(Relation heapRelation, extern void index_parallelrescan(IndexScanDesc scan); extern IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, - ParallelIndexScanDesc pscan); + ParallelIndexScanDesc pscan, + int prefetch_target, + int prefetch_reset); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); struct TupleTableSlot; @@ -230,4 +235,108 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); +/* + * XXX not sure it's the right place to define these callbacks etc. + */ +typedef void (*prefetcher_getrange_function) (IndexScanDesc scandesc, + ScanDirection direction, + int *start, int *end, + bool *reset); + +typedef BlockNumber (*prefetcher_getblock_function) (IndexScanDesc scandesc, + ScanDirection direction, + int index); + +/* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. Doesn't need to be perfectly accurate, but we + * aim to make false positives/negatives reasonably low. + */ +typedef struct PrefetchCacheEntry { + BlockNumber block; + uint64 request; +} PrefetchCacheEntry; + +/* + * Size of the cache of recently prefetched blocks - shouldn't be too + * small or too large. 1024 seems about right, it covers ~8MB of data. + * It's somewhat arbitrary, there's no particular formula saying it + * should not be higher/lower. + * + * The cache is structured as an array of small LRU caches, so the total + * size needs to be a multiple of LRU size. The LRU should be tiny to + * keep linear search cheap enough. + * + * XXX Maybe we could consider effective_cache_size or something? + */ +#define PREFETCH_LRU_SIZE 8 +#define PREFETCH_LRU_COUNT 128 +#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) + +/* + * Used to detect sequential patterns (and disable prefetching). + */ +#define PREFETCH_QUEUE_HISTORY 8 +#define PREFETCH_SEQ_PATTERN_BLOCKS 4 + + +typedef struct IndexPrefetchData +{ + /* + * XXX We need to disable this in some cases (e.g. when using index-only + * scans, we don't want to prefetch pages). Or maybe we should prefetch + * only pages that are not all-visible, that'd be even better. + */ + int prefetchTarget; /* how far we should be prefetching */ + int prefetchMaxTarget; /* maximum prefetching distance */ + int prefetchReset; /* reset to this distance on rescan */ + bool prefetchDone; /* did we get all TIDs from the index? */ + + /* runtime statistics */ + uint64 countAll; /* all prefetch requests */ + uint64 countPrefetch; /* actual prefetches */ + uint64 countSkipSequential; + uint64 countSkipCached; + + /* + * Queue of TIDs to prefetch. + * + * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler + * than dynamically adjusting for custom values. + */ + ItemPointerData queueItems[MAX_IO_CONCURRENCY]; + uint64 queueIndex; /* next TID to prefetch */ + uint64 queueStart; /* first valid TID in queue */ + uint64 queueEnd; /* first invalid (empty) TID in queue */ + + /* + * A couple of last prefetched blocks, used to check for certain access + * pattern and skip prefetching - e.g. for sequential access). + * + * XXX Separate from the main queue, because we only want to compare the + * block numbers, not the whole TID. In sequential access it's likely we + * read many items from each page, and we don't want to check many items + * (as that is much more expensive). + */ + BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; + uint64 blockIndex; /* index in the block (points to the first + * empty entry)*/ + + /* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. + */ + uint64 prefetchReqNumber; + PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; + +} IndexPrefetchData; + +#define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) +#define PREFETCH_QUEUE_EMPTY(p) ((p)->queueEnd == (p)->queueIndex) +#define PREFETCH_ENABLED(p) ((p) && ((p)->prefetchMaxTarget > 0)) +#define PREFETCH_FULL(p) ((p)->queueEnd - (p)->queueIndex == (p)->prefetchTarget) +#define PREFETCH_DONE(p) ((p) && ((p)->prefetchDone && PREFETCH_QUEUE_EMPTY(p))) +#define PREFETCH_ACTIVE(p) (PREFETCH_ENABLED(p) && !(p)->prefetchDone) +#define PREFETCH_BLOCK_INDEX(v) ((v) % PREFETCH_QUEUE_HISTORY) + #endif /* GENAM_H */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index d03360eac04..c119fe597d8 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -106,6 +106,12 @@ typedef struct IndexFetchTableData Relation rel; } IndexFetchTableData; +/* + * Forward declaration, defined in genam.h. + */ +typedef struct IndexPrefetchData IndexPrefetchData; +typedef struct IndexPrefetchData *IndexPrefetch; + /* * We use the same IndexScanDescData structure for both amgettuple-based * and amgetbitmap-based index scans. Some fields are only relevant in @@ -162,6 +168,9 @@ typedef struct IndexScanDescData bool *xs_orderbynulls; bool xs_recheckorderby; + /* prefetching state (or NULL if disabled) */ + IndexPrefetchData *xs_prefetch; + /* parallel index scan information, in shared memory */ struct ParallelIndexScanDescData *parallel_scan; } IndexScanDescData; diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index 87e5e2183bd..97dd3c2c421 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -33,6 +33,8 @@ typedef struct BufferUsage int64 local_blks_written; /* # of local disk blocks written */ int64 temp_blks_read; /* # of temp blocks read */ int64 temp_blks_written; /* # of temp blocks written */ + int64 blks_prefetch_rounds; /* # of prefetch rounds */ + int64 blks_prefetches; /* # of buffers prefetched */ instr_time blk_read_time; /* time spent reading blocks */ instr_time blk_write_time; /* time spent writing blocks */ instr_time temp_blk_read_time; /* time spent reading temp blocks */ -- 2.41.0 [text/x-patch] 0002-comments-and-minor-cleanup-20231012.patch (31.2K, 4-0002-comments-and-minor-cleanup-20231012.patch) download | inline diff: From 61b7123c6b3dbd716c6882716ce17239d38e0604 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Fri, 13 Oct 2023 22:34:40 +0200 Subject: [PATCH 2/4] comments and minor cleanup --- src/backend/access/gist/gistget.c | 1 + src/backend/access/heap/heapam_handler.c | 5 + src/backend/access/index/genam.c | 28 +- src/backend/access/index/indexam.c | 328 ++++++++++++++++------- src/backend/executor/nodeIndexscan.c | 17 ++ src/backend/replication/walsender.c | 2 - src/include/access/genam.h | 12 - 7 files changed, 273 insertions(+), 120 deletions(-) diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c index 3acfa762e7f..31349174280 100644 --- a/src/backend/access/gist/gistget.c +++ b/src/backend/access/gist/gistget.c @@ -677,6 +677,7 @@ gistgettuple(IndexScanDesc scan, ScanDirection dir) scan->xs_hitup = so->pageData[so->curPageData].recontup; so->curPageData++; + return true; } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 46c85751cf2..ca91bc5e878 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -750,6 +750,11 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ci_val[2]; int prefetch_target; + /* + * Get the prefetch target for the old tablespace (which is what we'll + * read using the index). We'll use it as a reset value too, although + * there should be no rescans for CLUSTER etc. + */ prefetch_target = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 230667f888b..6e3aa6bb1fd 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -126,7 +126,7 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys) scan->xs_hitup = NULL; scan->xs_hitupdesc = NULL; - /* set in each AM when applicable */ + /* Information used for asynchronous prefetching during index scans. */ scan->xs_prefetch = NULL; return scan; @@ -443,7 +443,18 @@ systable_beginscan(Relation heapRelation, elog(ERROR, "column is not in index"); } - /* no index prefetch for system catalogs */ + /* + * We don't do any prefetching on system catalogs, for two main reasons. + * + * Firstly, we usually do PK lookups, which makes prefetching pointles, + * or we often don't know how many rows to expect (and the numbers tend + * to be fairly low). So it's not clear it'd help. Furthermore, places + * that are sensitive tend to use syscache anyway. + * + * Secondly, we can't call get_tablespace_io_concurrency() because that + * does a sysscan internally, so it might lead to a cycle. We could use + * use effective_io_concurrency, but it doesn't seem worth it. + */ sysscan->iscan = index_beginscan(heapRelation, irel, snapshot, nkeys, 0, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); @@ -700,7 +711,18 @@ systable_beginscan_ordered(Relation heapRelation, elog(ERROR, "column is not in index"); } - /* no index prefetch for system catalogs */ + /* + * We don't do any prefetching on system catalogs, for two main reasons. + * + * Firstly, we usually do PK lookups, which makes prefetching pointles, + * or we often don't know how many rows to expect (and the numbers tend + * to be fairly low). So it's not clear it'd help. Furthermore, places + * that are sensitive tend to use syscache anyway. + * + * Secondly, we can't call get_tablespace_io_concurrency() because that + * does a sysscan internally, so it might lead to a cycle. We could use + * use effective_io_concurrency, but it doesn't seem worth it. + */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, snapshot, nkeys, 0, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 0b8f136f042..e45a3a89387 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -206,21 +206,30 @@ index_insert(Relation indexRelation, * * Caller must be holding suitable locks on the heap and the index. * - * prefetch_target determines if prefetching is requested for this index scan. - * We need to be able to disable this for two reasons. Firstly, we don't want - * to do prefetching for IOS (where we hope most of the heap pages won't be - * really needed. Secondly, we must prevent infinite loop when determining - * prefetch value for the tablespace - the get_tablespace_io_concurrency() - * does an index scan internally, which would result in infinite loop. So we - * simply disable prefetching in systable_beginscan(). - * - * XXX Maybe we should do prefetching even for catalogs, but then disable it - * when accessing TableSpaceRelationId. We still need the ability to disable - * this and catalogs are expected to be tiny, so prefetching is unlikely to - * make a difference. - * - * XXX The second reason doesn't really apply after effective_io_concurrency - * lookup moved to caller of index_beginscan. + * prefetch_target determines if prefetching is requested for this index scan, + * and how far ahead we want to prefetch + * + * prefetch_reset specifies the prefetch distance to start with on rescans (so + * that we don't ramp-up to prefetch_target and use that forever) + * + * Setting prefetch_target to 0 disables prefetching for the index scan. We do + * this for two reasons - for scans on system catalogs, and/or for cases where + * prefetching is expected to be pointless (like IOS). + * + * For system catalogs, we usually either scan by a PK value, or we we expect + * only few rows (or rather we don't know how many rows to expect). Also, we + * need to prevent infinite in the get_tablespace_io_concurrency() call - it + * does an index scan internally. So we simply disable prefetching for system + * catalogs. We could deal with this by picking a conservative static target + * (e.g. effective_io_concurrency, capped to something), but places that are + * performance sensitive likely use syscache anyway, and catalogs tend to be + * very small and hot. So we don't bother. + * + * For IOS, we expect to not need most heap pages (that's the whole point of + * IOS, actually), and prefetching them might lead to a lot of wasted I/O. + * + * XXX Not sure the infinite loop can still happen, now that the target lookup + * moved to callers of index_beginscan. */ IndexScanDesc index_beginscan(Relation heapRelation, @@ -264,8 +273,12 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); + /* + * No prefetch for bitmap index scans. In this case prefetching happens at + * the heapscan level. + */ scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, - 0, 0); /* no prefetch */ + 0, 0); /* * Save additional parameters into the scandesc. Everything else was set @@ -301,12 +314,13 @@ index_beginscan_internal(Relation indexRelation, /* * Tell the AM to open a scan. */ - scan = indexRelation->rd_indam->ambeginscan(indexRelation, nkeys, norderbys); + scan = indexRelation->rd_indam->ambeginscan(indexRelation, nkeys, + norderbys); /* Initialize information for parallel scan. */ scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; - /* with prefetching enabled, initialize the necessary state */ + /* With prefetching requested, initialize the prefetcher state. */ if (prefetch_target > 0) { IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); @@ -367,7 +381,7 @@ index_rescan(IndexScanDesc scan, prefetcher->queueEnd = 0; prefetcher->queueIndex = 0; prefetcher->prefetchDone = false; - + prefetcher->prefetchTarget = Min(prefetcher->prefetchTarget, prefetcher->prefetchReset); } @@ -399,7 +413,11 @@ index_endscan(IndexScanDesc scan) if (scan->xs_temp_snap) UnregisterSnapshot(scan->xs_snapshot); - /* If prefetching enabled, log prefetch stats. */ + /* + * If prefetching was enabled for this scan, log prefetch stats. + * + * FIXME This should really go to EXPLAIN ANALYZE instead. + */ if (scan->xs_prefetch) { IndexPrefetch prefetch = scan->xs_prefetch; @@ -554,8 +572,6 @@ index_parallelrescan(IndexScanDesc scan) * index_beginscan_parallel - join parallel index scan * * Caller must be holding suitable locks on the heap and the index. - * - * XXX See index_beginscan() for more comments on prefetch_target. */ IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, @@ -693,25 +709,31 @@ index_fetch_heap(IndexScanDesc scan, TupleTableSlot *slot) bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot) { - IndexPrefetch prefetch = scan->xs_prefetch; + IndexPrefetch prefetch = scan->xs_prefetch; /* for convenience */ for (;;) { - /* with prefetching enabled, accumulate enough TIDs into the prefetch */ + /* + * If the prefetching is still active (i.e. enabled and we still + * haven't finished reading TIDs from the scan), read enough TIDs into + * the queue until we hit the current target. + */ if (PREFETCH_ACTIVE(prefetch)) { - /* - * incrementally ramp up prefetch distance + /* + * Ramp up the prefetch distance incrementally. * - * XXX Intentionally done as first, so that with prefetching there's - * always at least one item in the queue. + * Intentionally done as first, before reading the TIDs into the + * queue, so that there's always at least one item. Otherwise we + * might get into a situation where we start with target=0 and no + * TIDs loaded. */ prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, - prefetch->prefetchMaxTarget); + prefetch->prefetchMaxTarget); /* - * get more TID while there is empty space in the queue (considering - * current prefetch target + * Now read TIDs from the index until the queue is full (with + * respect to the current prefetch target). */ while (!PREFETCH_FULL(prefetch)) { @@ -720,7 +742,10 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot * /* Time to fetch the next TID from the index */ tid = index_getnext_tid(scan, direction); - /* If we're out of index entries, we're done */ + /* + * If we're out of index entries, we're done (and we mark the + * the prefetcher as inactive). + */ if (tid == NULL) { prefetch->prefetchDone = true; @@ -732,22 +757,34 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot * prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; prefetch->queueEnd++; + /* + * Issue the actuall prefetch requests for the new TID. + * + * FIXME For IOS, this should prefetch only pages that are not + * fully visible. + */ index_prefetch(scan, tid); } } if (!scan->xs_heap_continue) { + /* + * With prefetching enabled (even if we already finished reading + * all TIDs from the index scan), we need to return a TID from the + * queue. Otherwise, we just get the next TID from the scan + * directly. + */ if (PREFETCH_ENABLED(prefetch)) { - /* prefetching enabled, but reached the end and queue empty */ + /* Did we reach the end of the scan and the queue is empty? */ if (PREFETCH_DONE(prefetch)) break; scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; prefetch->queueIndex++; } - else /* not prefetching, just do the regular work */ + else /* not prefetching, just do the regular work */ { ItemPointer tid; @@ -1114,15 +1151,49 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, } /* - * Add the block to the tiny top-level queue (LRU), and check if the block - * is in a sequential pattern. + * index_prefetch_is_sequential + * Track the block number and check if the I/O pattern is sequential, + * or if the same block was just prefetched. + * + * Prefetching is cheap, but for some access patterns the benefits are small + * compared to the extra overhead. In particular, for sequential access the + * read-ahead performed by the OS is very effective/efficient. Doing more + * prefetching is just increasing the costs. + * + * This tries to identify simple sequential patterns, so that we can skip + * the prefetching request. This is implemented by having a small queue + * of block numbers, and checking it before prefetching another block. + * + * We look at the preceding PREFETCH_SEQ_PATTERN_BLOCKS blocks, and see if + * they are sequential. We also check if the block is the same as the last + * request (which is not sequential). + * + * Note that the main prefetch queue is not really useful for this, as it + * stores TIDs while we care about block numbers. Consider a sorted table, + * with a perfectly sequential pattern when accessed through an index. Each + * heap page may have dozens of TIDs, but we need to check block numbers. + * We could keep enough TIDs to cover enough blocks, but then we also need + * to walk those when checking the pattern (in hot path). + * + * So instead, we maintain a small separate queue of block numbers, and we use + * this instead. + * + * Returns true if the block is in a sequential pattern (and so should not be + * prefetched), or false (not sequential, should be prefetched). + * + * XXX The name is a bit misleading, as it also adds the block number to the + * block queue and checks if the block is the same as the last one (which + * does not require a sequential pattern). */ static bool index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) { - int idx; + int idx; - /* If the queue is empty, just store the block and we're done. */ + /* + * If the block queue is empty, just store the block and we're done (it's + * neither a sequential pattern, neither recently prefetched block). + */ if (prefetch->blockIndex == 0) { prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; @@ -1131,30 +1202,66 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) } /* - * Otherwise, check if it's the same as the immediately preceding block (we - * don't want to prefetch the same block over and over.) + * Check if it's the same as the immediately preceding block. We don't + * want to prefetch the same block over and over (which would happen for + * well correlated indexes). + * + * In principle we could rely on index_prefetch_add_cache doing this using + * the full cache, but this check is much cheaper and we need to look at + * the preceding block anyway, so we just do it. + * + * XXX Notice we haven't added the block to the block queue yet, and there + * is a preceding block (i.e. blockIndex-1 is valid). */ if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) return true; - /* Not the same block, so add it to the queue. */ + /* + * Add the block number to the queue. + * + * We do this before checking if the pattern, because we want to know + * about the block even if we end up skipping the prefetch. Otherwise we'd + * not be able to detect longer sequential pattens - we'd skip one block + * but then fail to skip the next couple blocks even in a perfect + * sequential pattern. This ocillation might even prevent the OS + * read-ahead from kicking in. + */ prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; prefetch->blockIndex++; - /* check sequential patter a couple requests back */ + /* + * Check if the last couple blocks are in a sequential pattern. We look + * for a sequential pattern of PREFETCH_SEQ_PATTERN_BLOCKS (4 by default), + * so we look for patterns of 5 pages (40kB) including the new block. + * + * XXX Perhaps this should be tied to effective_io_concurrency somehow? + * + * XXX Could it be harmful that we read the queue backwards? Maybe memory + * prefetching works better for the forward direction? + */ for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) { - /* not enough requests to confirm a sequential pattern */ + /* + * Are there enough requests to confirm a sequential pattern? We only + * consider something to be sequential after finding a sequence of + * PREFETCH_SEQ_PATTERN_BLOCKS blocks. + * + * FIXME Better to move this outside the loop. + */ if (prefetch->blockIndex < i) return false; /* - * index of the already requested buffer (-1 because we already - * incremented the index when adding the block to the queue) + * Calculate index of the earlier block (we need to do -1 as we + * already incremented the index when adding the new block to the + * queue). */ idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); - /* */ + /* + * For a sequential pattern, blocks "k" step ago needs to have block + * number by "k" smaller compared to the current block. + */ if (prefetch->blockItems[idx] != (block - i)) return false; } @@ -1164,30 +1271,34 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) /* * index_prefetch_add_cache - * Add a block to the cache, return true if it was recently prefetched. + * Add a block to the cache, check if it was recently prefetched. + * + * We don't want to prefetch blocks that we already prefetched recently. It's + * cheap but not free, and the overhead may have measurable impact. * - * When checking a block, we need to check if it was recently prefetched, - * where recently means within PREFETCH_CACHE_SIZE requests. This check - * needs to be very cheap, even with fairly large caches (hundreds of - * entries). The cache does not need to be perfect, we can accept false - * positives/negatives, as long as the rate is reasonably low. We also - * need to expire entries, so that only "recent" requests are remembered. + * This check needs to be very cheap, even with fairly large caches (hundreds + * of entries, see PREFETCH_CACHE_SIZE). * - * A queue would allow expiring the requests, but checking if a block was - * prefetched would be expensive (linear search for longer queues). Another - * option would be a hash table, but that has issues with expiring entries - * cheaply (which usually degrades the hash table). + * A simple queue would allow expiring the requests, but checking if it + * contains a particular block prefetched would be expensive (linear search). + * Another option would be a simple hash table, which has fast lookup but + * does not allow expiring entries cheaply. * - * So we use a cache that is organized as multiple small LRU caches. Each + * The cache does not need to be perfect, we can accept false + * positives/negatives, as long as the rate is reasonably low. We also need + * to expire entries, so that only "recent" requests are remembered. + * + * We use a hybrid cache that is organized as many small LRU caches. Each * block is mapped to a particular LRU by hashing (so it's a bit like a - * hash table), and each LRU is tiny (e.g. 8 entries). The LRU only keeps - * the most recent requests (for that particular LRU). + * hash table). The LRU caches are tiny (e.g. 8 entries), and the expiration + * happens at the level of a single LRU (by tracking only the 8 most recent requests). * - * This allows quick searches and expiration, with false negatives (when - * a particular LRU has too many collisions). + * This allows quick searches and expiration, but with false negatives (when a + * particular LRU has too many collisions, we may evict entries that are more + * recent than some other LRU). * * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 - * prefetch request in total. + * prefetch request in total (these are the default parameters.) * * The recency is determined using a prefetch counter, incremented every * time we end up prefetching a block. The counter is uint64, so it should @@ -1197,33 +1308,39 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) * and then linearly search if the tiny LRU has entry for the same block * and request less than PREFETCH_CACHE_SIZE ago. * - * At the same time, we either update the entry (for the same block) if + * At the same time, we either update the entry (for the queried block) if * found, or replace the oldest/empty entry. * * If the block was not recently prefetched (i.e. we want to prefetch it), * we increment the counter. + * + * Returns true if the block was recently prefetched (and thus we don't + * need to prefetch it again), or false (should do a prefetch). + * + * XXX It's a bit confusing these return values are inverse compared to + * what index_prefetch_is_sequential does. */ static bool index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) { PrefetchCacheEntry *entry; - /* calculate which LRU to use */ + /* map the block number the the LRU */ int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; - /* entry to (maybe) use for this block request */ + /* age/index of the oldest entry in the LRU, to maybe use */ uint64 oldestRequest = PG_UINT64_MAX; int oldestIndex = -1; /* * First add the block to the (tiny) top-level LRU cache and see if it's - * part of a sequential pattern. In this case we just ignore the block - * and don't prefetch it - we expect read-ahead to do a better job. + * part of a sequential pattern. In this case we just ignore the block and + * don't prefetch it - we expect read-ahead to do a better job. * - * XXX Maybe we should still add the block to the later cache, in case - * we happen to access it later? That might help if we first scan a lot - * of the table sequentially, and then randomly. Not sure that's very - * likely with index access, though. + * XXX Maybe we should still add the block to the hybrid cache, in case we + * happen to access it later? That might help if we first scan a lot of + * the table sequentially, and then randomly. Not sure that's very likely + * with index access, though. */ if (index_prefetch_is_sequential(prefetch, block)) { @@ -1231,7 +1348,11 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) return true; } - /* see if we already have prefetched this block (linear search of LRU) */ + /* + * See if we recently prefetched this block - we simply scan the LRU + * linearly. While doing that, we also track the oldest entry, so that we + * know where to put the block if we don't find a matching entry. + */ for (int i = 0; i < PREFETCH_LRU_SIZE; i++) { entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; @@ -1243,14 +1364,18 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) oldestIndex = i; } - /* Request numbers are positive, so 0 means "unused". */ + /* + * If the entry is unused (identified by request being set to 0), + * we're done. Notice the field is uint64, so empty entry is + * guaranteed to be the oldest one. + */ if (entry->request == 0) continue; /* Is this entry for the same block as the current request? */ if (entry->block == block) { - bool prefetched; + bool prefetched; /* * Is the old request sufficiently recent? If yes, we treat the @@ -1259,7 +1384,7 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) * XXX We do add the cache size to the request in order not to * have issues with uint64 underflows. */ - prefetched = (entry->request + PREFETCH_CACHE_SIZE >= prefetch->prefetchReqNumber); + prefetched = ((entry->request + PREFETCH_CACHE_SIZE) >= prefetch->prefetchReqNumber); /* Update the request number. */ entry->request = ++prefetch->prefetchReqNumber; @@ -1276,6 +1401,7 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) */ Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); + /* FIXME do a nice macro */ entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; entry->block = block; @@ -1286,32 +1412,31 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) } /* - * Do prefetching, and gradually increase the prefetch distance. - * - * XXX This is limited to a single index page (because that's where we get - * currPos.items from). But index tuples are typically very small, so there - * should be quite a bit of stuff to prefetch (especially with deduplicated - * indexes, etc.). Does not seem worth reworking the index access to allow - * more aggressive prefetching, it's best effort. + * index_prefetch + * Prefetch the TID, unless it's sequential or recently prefetched. * * XXX Some ideas how to auto-tune the prefetching, so that unnecessary * prefetching does not cause significant regressions (e.g. for nestloop - * with inner index scan). We could track number of index pages visited - * and index tuples returned, to calculate avg tuples / page, and then - * use that to limit prefetching after switching to a new page (instead - * of just using prefetchMaxTarget, which can get much larger). + * with inner index scan). We could track number of rescans and number of + * items (TIDs) actually returned from the scan. Then we could calculate + * rows / rescan and use that to clamp prefetch target. * - * XXX Obviously, another option is to use the planner estimates - we know - * how many rows we're expected to fetch (on average, assuming the estimates - * are reasonably accurate), so why not to use that. And maybe combine it - * with the auto-tuning based on runtime statistics, described above. + * That'd help with cases when a scan matches only very few rows, far less + * than the prefetchTarget, because the unnecessary prefetches are wasted + * I/O. Imagine a LIMIT on top of index scan, or something like that. + * + * Another option is to use the planner estimates - we know how many rows we're + * expecting to fetch (on average, assuming the estimates are reasonably + * accurate), so why not to use that? + * + * Of course, we could/should combine these two approaches. * * XXX The prefetching may interfere with the patch allowing us to evaluate * conditions on the index tuple, in which case we may not need the heap * tuple. Maybe if there's such filter, we should prefetch only pages that * are not all-visible (and the same idea would also work for IOS), but * it also makes the indexing a bit "aware" of the visibility stuff (which - * seems a bit wrong). Also, maybe we should consider the filter selectivity + * seems a somewhat wrong). Also, maybe we should consider the filter selectivity * (if the index-only filter is expected to eliminate only few rows, then * the vm check is pointless). Maybe this could/should be auto-tuning too, * i.e. we could track how many heap tuples were needed after all, and then @@ -1324,13 +1449,13 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) static void index_prefetch(IndexScanDesc scan, ItemPointer tid) { - IndexPrefetch prefetch = scan->xs_prefetch; - BlockNumber block; + IndexPrefetch prefetch = scan->xs_prefetch; + BlockNumber block; /* - * No heap relation means bitmap index scan, which does prefetching at - * the bitmap heap scan, so no prefetch here (we can't do it anyway, - * without the heap) + * No heap relation means bitmap index scan, which does prefetching at the + * bitmap heap scan, so no prefetch here (we can't do it anyway, without + * the heap) * * XXX But in this case we should have prefetchMaxTarget=0, because in * index_bebinscan_bitmap() we disable prefetching. So maybe we should @@ -1339,9 +1464,6 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid) if (!prefetch) return; - /* was it initialized correctly? */ - // Assert(prefetch->prefetchIndex != -1); - /* * If we got here, prefetching is enabled and it's a node that supports * prefetching (i.e. it can't be a bitmap index scan). @@ -1355,9 +1477,9 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid) /* * Do not prefetch the same block over and over again, * - * This happens e.g. for clustered or naturally correlated indexes - * (fkey to a sequence ID). It's not expensive (the block is in page - * cache already, so no I/O), but it's not free either. + * This happens e.g. for clustered or naturally correlated indexes (fkey + * to a sequence ID). It's not expensive (the block is in page cache + * already, so no I/O), but it's not free either. */ if (!index_prefetch_add_cache(prefetch, block)) { diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 185ff0f1449..9796f8b979c 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -1710,6 +1710,11 @@ ExecIndexScanInitializeDSM(IndexScanState *node, * essentially just effective_io_concurrency for the table (or the * tablespace it's in). * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * * XXX Maybe reduce the value with parallel workers? */ heapRel = node->ss.ss_currentRelation; @@ -1770,6 +1775,18 @@ ExecIndexScanInitializeWorker(IndexScanState *node, int prefetch_target; int prefetch_reset; + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ heapRel = node->ss.ss_currentRelation; prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 47093cc9cf1..e250b0567eb 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1131,8 +1131,6 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) need_full_snapshot = true; } - elog(LOG, "slot = %s need_full_snapshot = %d", cmd->slotname, need_full_snapshot); - ctx = CreateInitDecodingContext(cmd->plugin, NIL, need_full_snapshot, InvalidXLogRecPtr, XL_ROUTINE(.page_read = logical_read_xlog_page, diff --git a/src/include/access/genam.h b/src/include/access/genam.h index b814af4b2f6..907ab886d3e 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -235,18 +235,6 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); -/* - * XXX not sure it's the right place to define these callbacks etc. - */ -typedef void (*prefetcher_getrange_function) (IndexScanDesc scandesc, - ScanDirection direction, - int *start, int *end, - bool *reset); - -typedef BlockNumber (*prefetcher_getblock_function) (IndexScanDesc scandesc, - ScanDirection direction, - int index); - /* * Cache of recently prefetched blocks, organized as a hash table of * small LRU caches. Doesn't need to be perfectly accurate, but we -- 2.41.0 [text/x-patch] 0003-remove-prefetch_reset-20231012.patch (17.5K, 5-0003-remove-prefetch_reset-20231012.patch) download | inline diff: From 61b13cc9a2a0445d6ab992520a945437cd0f275c Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Fri, 13 Oct 2023 23:04:39 +0200 Subject: [PATCH 3/4] remove prefetch_reset --- src/backend/access/heap/heapam_handler.c | 6 +-- src/backend/access/index/genam.c | 4 +- src/backend/access/index/indexam.c | 38 +++++++---------- src/backend/executor/execIndexing.c | 2 +- src/backend/executor/execReplication.c | 2 +- src/backend/executor/nodeIndexonlyscan.c | 6 +-- src/backend/executor/nodeIndexscan.c | 53 ++++++++++-------------- src/backend/utils/adt/selfuncs.c | 3 +- src/include/access/genam.h | 6 +-- src/include/access/relscan.h | 4 +- 10 files changed, 52 insertions(+), 72 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index ca91bc5e878..89474078951 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -748,14 +748,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, PROGRESS_CLUSTER_INDEX_RELID }; int64 ci_val[2]; - int prefetch_target; + int prefetch_max; /* * Get the prefetch target for the old tablespace (which is what we'll * read using the index). We'll use it as a reset value too, although * there should be no rescans for CLUSTER etc. */ - prefetch_target = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); + prefetch_max = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; @@ -765,7 +765,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tableScan = NULL; heapScan = NULL; indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0, - prefetch_target, prefetch_target); + prefetch_max); index_rescan(indexScan, NULL, 0, NULL, 0); } else diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 6e3aa6bb1fd..d45a209ee3a 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -456,7 +456,7 @@ systable_beginscan(Relation heapRelation, * use effective_io_concurrency, but it doesn't seem worth it. */ sysscan->iscan = index_beginscan(heapRelation, irel, - snapshot, nkeys, 0, 0, 0); + snapshot, nkeys, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; } @@ -724,7 +724,7 @@ systable_beginscan_ordered(Relation heapRelation, * use effective_io_concurrency, but it doesn't seem worth it. */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, - snapshot, nkeys, 0, 0, 0); + snapshot, nkeys, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index e45a3a89387..8c56acfd638 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -109,7 +109,7 @@ do { \ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, ParallelIndexScanDesc pscan, bool temp_snap, - int prefetch_target, int prefetch_reset); + int prefetch_max); static void index_prefetch(IndexScanDesc scan, ItemPointer tid); @@ -206,13 +206,10 @@ index_insert(Relation indexRelation, * * Caller must be holding suitable locks on the heap and the index. * - * prefetch_target determines if prefetching is requested for this index scan, + * prefetch_max determines if prefetching is requested for this index scan, * and how far ahead we want to prefetch * - * prefetch_reset specifies the prefetch distance to start with on rescans (so - * that we don't ramp-up to prefetch_target and use that forever) - * - * Setting prefetch_target to 0 disables prefetching for the index scan. We do + * Setting prefetch_max to 0 disables prefetching for the index scan. We do * this for two reasons - for scans on system catalogs, and/or for cases where * prefetching is expected to be pointless (like IOS). * @@ -236,14 +233,14 @@ index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, int nkeys, int norderbys, - int prefetch_target, int prefetch_reset) + int prefetch_max) { IndexScanDesc scan; Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false, - prefetch_target, prefetch_reset); + scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, + NULL, false, prefetch_max); /* * Save additional parameters into the scandesc. Everything else was set @@ -273,12 +270,8 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); - /* - * No prefetch for bitmap index scans. In this case prefetching happens at - * the heapscan level. - */ - scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, - 0, 0); + /* No prefetch in bitmap scans, prefetch is done by the heap scan. */ + scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, 0); /* * Save additional parameters into the scandesc. Everything else was set @@ -296,7 +289,7 @@ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, ParallelIndexScanDesc pscan, bool temp_snap, - int prefetch_target, int prefetch_reset) + int prefetch_max) { IndexScanDesc scan; @@ -321,7 +314,7 @@ index_beginscan_internal(Relation indexRelation, scan->xs_temp_snap = temp_snap; /* With prefetching requested, initialize the prefetcher state. */ - if (prefetch_target > 0) + if (prefetch_max > 0) { IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); @@ -330,8 +323,7 @@ index_beginscan_internal(Relation indexRelation, prefetcher->queueEnd = 0; prefetcher->prefetchTarget = 0; - prefetcher->prefetchMaxTarget = prefetch_target; - prefetcher->prefetchReset = prefetch_reset; + prefetcher->prefetchMaxTarget = prefetch_max; scan->xs_prefetch = prefetcher; } @@ -382,8 +374,8 @@ index_rescan(IndexScanDesc scan, prefetcher->queueIndex = 0; prefetcher->prefetchDone = false; - prefetcher->prefetchTarget = Min(prefetcher->prefetchTarget, - prefetcher->prefetchReset); + /* restart the incremental ramp-up */ + prefetcher->prefetchTarget = 0; } } @@ -576,7 +568,7 @@ index_parallelrescan(IndexScanDesc scan) IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, ParallelIndexScanDesc pscan, - int prefetch_target, int prefetch_reset) + int prefetch_max) { Snapshot snapshot; IndexScanDesc scan; @@ -585,7 +577,7 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, snapshot = RestoreSnapshot(pscan->ps_snapshot_data); RegisterSnapshot(snapshot); scan = index_beginscan_internal(indexrel, nkeys, norderbys, snapshot, - pscan, true, prefetch_target, prefetch_reset); + pscan, true, prefetch_max); /* * Save additional parameters into the scandesc. Everything else was set diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 09418f715fa..eae1d8f9233 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -773,7 +773,7 @@ check_exclusion_or_unique_constraint(Relation heap, Relation index, retry: conflict = false; found_self = false; - index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0, 0); + index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0); index_rescan(index_scan, scankeys, indnkeyatts, NULL, 0); while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot)) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index f3e1a8d22a4..91676ccff95 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -210,7 +210,7 @@ RelationFindReplTupleByIndex(Relation rel, Oid idxoid, * probably using a PK / UNIQUE index, so does not seem worth it. If we * reconsider this, calclate prefetch_target like in nodeIndexscan. */ - scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0, 0); + scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0); retry: found = false; diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 75b44db33c6..7c27913502c 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -100,7 +100,7 @@ IndexOnlyNext(IndexOnlyScanState *node) estate->es_snapshot, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - 0, 0); /* no index prefetch for IOS */ + 0); /* no prefetching for IOS */ node->ioss_ScanDesc = scandesc; @@ -667,7 +667,7 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, piscan, - 0, 0); /* no index prefetch for IOS */ + 0); /* no prefetching for IOS */ node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -713,7 +713,7 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, piscan, - 0, 0); /* no index prefetch for IOS */ + 0); /* no prefetching for IOS */ node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 9796f8b979c..a5f5394ef49 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -105,12 +105,11 @@ IndexNext(IndexScanState *node) if (scandesc == NULL) { - int prefetch_target; - int prefetch_reset; + int prefetch_max; /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the + * Determine number of heap pages to prefetch for this index scan. This + * is essentially just effective_io_concurrency for the table (or the * tablespace it's in). * * XXX Should this also look at plan.plan_rows and maybe cap the target @@ -118,8 +117,8 @@ IndexNext(IndexScanState *node) * just reset to that value during prefetching, after reading the next * index page (or rather after rescan)? */ - prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); - prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); /* * We reach here if the index scan is not parallel, or if we're @@ -130,8 +129,7 @@ IndexNext(IndexScanState *node) estate->es_snapshot, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - prefetch_target, - prefetch_reset); + prefetch_max); node->iss_ScanDesc = scandesc; @@ -197,6 +195,7 @@ IndexNextWithReorder(IndexScanState *node) Datum *lastfetched_vals; bool *lastfetched_nulls; int cmp; + Relation heapRel = node->ss.ss_currentRelation; estate = node->ss.ps.state; @@ -218,9 +217,7 @@ IndexNextWithReorder(IndexScanState *node) if (scandesc == NULL) { - Relation heapRel = node->ss.ss_currentRelation; - int prefetch_target; - int prefetch_reset; + int prefetch_max; /* * Determine number of heap pages to prefetch for this index. This is @@ -232,8 +229,8 @@ IndexNextWithReorder(IndexScanState *node) * just reset to that value during prefetching, after reading the next * index page (or rather after rescan)? */ - prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); - prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); /* * We reach here if the index scan is not parallel, or if we're @@ -244,8 +241,7 @@ IndexNextWithReorder(IndexScanState *node) estate->es_snapshot, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - prefetch_target, - prefetch_reset); + prefetch_max); node->iss_ScanDesc = scandesc; @@ -1701,9 +1697,8 @@ ExecIndexScanInitializeDSM(IndexScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; - Relation heapRel; - int prefetch_target; - int prefetch_reset; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; /* * Determine number of heap pages to prefetch for this index. This is @@ -1717,10 +1712,9 @@ ExecIndexScanInitializeDSM(IndexScanState *node, * * XXX Maybe reduce the value with parallel workers? */ - heapRel = node->ss.ss_currentRelation; - prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); - prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -1734,8 +1728,7 @@ ExecIndexScanInitializeDSM(IndexScanState *node, node->iss_NumScanKeys, node->iss_NumOrderByKeys, piscan, - prefetch_target, - prefetch_reset); + prefetch_max); /* * If no run-time keys to calculate or they are ready, go ahead and pass @@ -1771,9 +1764,8 @@ ExecIndexScanInitializeWorker(IndexScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; - Relation heapRel; - int prefetch_target; - int prefetch_reset; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; /* * Determine number of heap pages to prefetch for this index. This is @@ -1787,10 +1779,8 @@ ExecIndexScanInitializeWorker(IndexScanState *node, * * XXX Maybe reduce the value with parallel workers? */ - heapRel = node->ss.ss_currentRelation; - - prefetch_target = get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace); - prefetch_reset = Min(prefetch_target, node->ss.ps.plan->plan_rows); + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->iss_ScanDesc = @@ -1799,8 +1789,7 @@ ExecIndexScanInitializeWorker(IndexScanState *node, node->iss_NumScanKeys, node->iss_NumOrderByKeys, piscan, - prefetch_target, - prefetch_reset); + prefetch_max); /* * If no run-time keys to calculate or they are ready, go ahead and pass diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 0b02b6265d0..52e3aaaf20a 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -6216,9 +6216,10 @@ get_actual_variable_endpoint(Relation heapRel, InitNonVacuumableSnapshot(SnapshotNonVacuumable, GlobalVisTestFor(heapRel)); + /* XXX Maybe should do prefetching using the default prefetch parameters? */ index_scan = index_beginscan(heapRel, indexRel, &SnapshotNonVacuumable, - 1, 0, 0, 0); /* XXX maybe do prefetch? */ + 1, 0, 0); /* Set it up for index-only scan */ index_scan->xs_want_itup = true; index_rescan(index_scan, scankeys, 1, NULL, 0); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 907ab886d3e..ceb6279b0b0 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -154,8 +154,7 @@ extern IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, int nkeys, int norderbys, - int prefetch_target, - int prefetch_reset); + int prefetch_max); extern IndexScanDesc index_beginscan_bitmap(Relation indexRelation, Snapshot snapshot, int nkeys); @@ -173,8 +172,7 @@ extern void index_parallelrescan(IndexScanDesc scan); extern IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, ParallelIndexScanDesc pscan, - int prefetch_target, - int prefetch_reset); + int prefetch_max); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); struct TupleTableSlot; diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index c119fe597d8..231a30ecc46 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -107,7 +107,7 @@ typedef struct IndexFetchTableData } IndexFetchTableData; /* - * Forward declaration, defined in genam.h. + * Forward declarations, defined in genam.h. */ typedef struct IndexPrefetchData IndexPrefetchData; typedef struct IndexPrefetchData *IndexPrefetch; @@ -168,7 +168,7 @@ typedef struct IndexScanDescData bool *xs_orderbynulls; bool xs_recheckorderby; - /* prefetching state (or NULL if disabled) */ + /* prefetching state (or NULL if disabled for this scan) */ IndexPrefetchData *xs_prefetch; /* parallel index scan information, in shared memory */ -- 2.41.0 [text/x-patch] 0004-PoC-prefetch-for-IOS-20231012.patch (11.3K, 6-0004-PoC-prefetch-for-IOS-20231012.patch) download | inline diff: From 047eda4fa39e5b37b5beaaa3f24195a8d7aa6a5e Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Sat, 14 Oct 2023 00:13:26 +0200 Subject: [PATCH 4/4] PoC prefetch for IOS --- src/backend/access/index/indexam.c | 140 ++++++++++++++++++++++- src/backend/executor/nodeIndexonlyscan.c | 63 +++++++++- src/include/access/genam.h | 2 + 3 files changed, 195 insertions(+), 10 deletions(-) diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 8c56acfd638..4ae4e867770 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -49,6 +49,7 @@ #include "access/relscan.h" #include "access/tableam.h" #include "access/transam.h" +#include "access/visibilitymap.h" #include "access/xlog.h" #include "catalog/index.h" #include "catalog/pg_amproc.h" @@ -111,7 +112,7 @@ static IndexScanDesc index_beginscan_internal(Relation indexRelation, ParallelIndexScanDesc pscan, bool temp_snap, int prefetch_max); -static void index_prefetch(IndexScanDesc scan, ItemPointer tid); +static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible); /* ---------------------------------------------------------------- @@ -755,7 +756,7 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot * * FIXME For IOS, this should prefetch only pages that are not * fully visible. */ - index_prefetch(scan, tid); + index_prefetch(scan, tid, false); } } @@ -1439,7 +1440,7 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) * in BTScanPosData.nextPage. */ static void -index_prefetch(IndexScanDesc scan, ItemPointer tid) +index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) { IndexPrefetch prefetch = scan->xs_prefetch; BlockNumber block; @@ -1462,10 +1463,36 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid) */ Assert(scan->heapRelation); - prefetch->countAll++; - block = ItemPointerGetBlockNumber(tid); + /* + * When prefetching for IOS, we want to only prefetch pages that are not + * marked as all-visible (because not fetching all-visible pages is the + * point of IOS). + * + * XXX This is not great, because it releases the VM buffer for each TID + * we consider to prefetch. We should reuse that somehow, similar to the + * actual IOS code. Ideally, we should use the same ioss_VMBuffer (if + * we can propagate it here). Or at least do it for a bulk of prefetches, + * although that's not very useful - after the ramp-up we will prefetch + * the pages one by one anyway. + */ + if (skip_all_visible) + { + bool all_visible; + Buffer vmbuffer = InvalidBuffer; + + all_visible = VM_ALL_VISIBLE(scan->heapRelation, + block, + &vmbuffer); + + if (vmbuffer != InvalidBuffer) + ReleaseBuffer(vmbuffer); + + if (all_visible) + return; + } + /* * Do not prefetch the same block over and over again, * @@ -1480,4 +1507,107 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid) PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); pgBufferUsage.blks_prefetches++; } + + prefetch->countAll++; +} + +/* ---------------- + * index_getnext_tid_prefetch - get the next TID from a scan + * + * The result is the next TID satisfying the scan keys, + * or NULL if no more matching tuples exist. + * + * FIXME not sure this handles xs_heapfetch correctly. + * ---------------- + */ +ItemPointer +index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) +{ + IndexPrefetch prefetch = scan->xs_prefetch; /* for convenience */ + + /* + * If the prefetching is still active (i.e. enabled and we still + * haven't finished reading TIDs from the scan), read enough TIDs into + * the queue until we hit the current target. + */ + if (PREFETCH_ACTIVE(prefetch)) + { + /* + * Ramp up the prefetch distance incrementally. + * + * Intentionally done as first, before reading the TIDs into the + * queue, so that there's always at least one item. Otherwise we + * might get into a situation where we start with target=0 and no + * TIDs loaded. + */ + prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, + prefetch->prefetchMaxTarget); + + /* + * Now read TIDs from the index until the queue is full (with + * respect to the current prefetch target). + */ + while (!PREFETCH_FULL(prefetch)) + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* + * If we're out of index entries, we're done (and we mark the + * the prefetcher as inactive). + */ + if (tid == NULL) + { + prefetch->prefetchDone = true; + break; + } + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; + prefetch->queueEnd++; + + /* + * Issue the actuall prefetch requests for the new TID. + * + * XXX index_getnext_tid_prefetch is only called for IOS (for now), + * so skip prefetching of all-visible pages. + */ + index_prefetch(scan, tid, true); + } + } + + /* + * With prefetching enabled (even if we already finished reading + * all TIDs from the index scan), we need to return a TID from the + * queue. Otherwise, we just get the next TID from the scan + * directly. + */ + if (PREFETCH_ENABLED(prefetch)) + { + /* Did we reach the end of the scan and the queue is empty? */ + if (PREFETCH_DONE(prefetch)) + return NULL; + + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; + prefetch->queueIndex++; + } + else /* not prefetching, just do the regular work */ + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* If we're out of index entries, we're done */ + if (tid == NULL) + return NULL; + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + } + + /* Return the TID of the tuple we found. */ + return &scan->xs_heaptid; } diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 7c27913502c..855afd5ba76 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -43,7 +43,7 @@ #include "storage/predicate.h" #include "utils/memutils.h" #include "utils/rel.h" - +#include "utils/spccache.h" static TupleTableSlot *IndexOnlyNext(IndexOnlyScanState *node); static void StoreIndexTuple(TupleTableSlot *slot, IndexTuple itup, @@ -65,6 +65,7 @@ IndexOnlyNext(IndexOnlyScanState *node) IndexScanDesc scandesc; TupleTableSlot *slot; ItemPointer tid; + Relation heapRel = node->ss.ss_currentRelation; /* * extract necessary information from index scan node @@ -83,6 +84,23 @@ IndexOnlyNext(IndexOnlyScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index only scan is not parallel, or if we're * serially executing an index only scan that was planned to be @@ -100,7 +118,7 @@ IndexOnlyNext(IndexOnlyScanState *node) estate->es_snapshot, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - 0); /* no prefetching for IOS */ + prefetch_max); node->ioss_ScanDesc = scandesc; @@ -124,7 +142,7 @@ IndexOnlyNext(IndexOnlyScanState *node) /* * OK, now that we have what we need, fetch the next tuple. */ - while ((tid = index_getnext_tid(scandesc, direction)) != NULL) + while ((tid = index_getnext_tid_prefetch(scandesc, direction)) != NULL) { bool tuple_from_heap = false; @@ -654,6 +672,24 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->ioss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -667,7 +703,7 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, piscan, - 0); /* no prefetching for IOS */ + prefetch_max); node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -705,6 +741,23 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->ioss_ScanDesc = @@ -713,7 +766,7 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, piscan, - 0); /* no prefetching for IOS */ + prefetch_max); node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/include/access/genam.h b/src/include/access/genam.h index ceb6279b0b0..6e92758ced5 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -175,6 +175,8 @@ extern IndexScanDesc index_beginscan_parallel(Relation heaprel, int prefetch_max); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); +extern ItemPointer index_getnext_tid_prefetch(IndexScanDesc scan, + ScanDirection direction); struct TupleTableSlot; extern bool index_fetch_heap(IndexScanDesc scan, struct TupleTableSlot *slot); extern bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, -- 2.41.0 ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-11-24 16:25 Tomas Vondra <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Tomas Vondra @ 2023-11-24 16:25 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> Hi, Here's a new WIP version of the patch set adding prefetching to indexes, exploring a couple alternative approaches. After the patch 2023/10/16 version, I happened to have an off-list discussion with Andres, and he suggested to try a couple things, and there's a couple more things I tried on my own too. Attached is the patch series starting with the 2023/10/16 patch, and then trying different things in separate patches (discussed later). As usual, there's also a bunch of benchmark results - due to size I'm unable to attach all of them here (the PDFs are pretty large), but you can find them at (with all the scripts etc.): https://github.com/tvondra/index-prefetch-tests/tree/master/2023-11-23 I'll attach only a couple small PNG with highlighted speedup/regression patterns, but it's unreadable and more of a pointer to the PDF. A quick overview of the patches ------------------------------- v20231124-0001-prefetch-2023-10-16.patch - same as the October 16 patch, with only minor comment tweaks v20231124-0002-rely-on-PrefetchBuffer-instead-of-custom-c.patch - removes custom cache of recently prefetched blocks, replaces it simply by calling PrefetchBuffer (which check shared buffers) v20231124-0003-check-page-cache-using-preadv2.patch - adds a check using preadv2(RWF_NOWAIT) to check if the whole page is in page cache v20231124-0004-reintroduce-the-LRU-cache-of-recent-blocks.patch - adds back a small LRU cache to identify sequential patterns (based on benchmarks of 0002/0003 patches) v20231124-0005-hold-the-vm-buffer-for-IOS-prefetching.patch v20231124-0006-poc-reuse-vm-information.patch - optimizes the visibilitymap handling when prefetching for IOS (to deal with overhead in the all-visible cases) by v20231124-0007-20231016-reworked.patch - returns back to the 20231016 patch, but this time with the VM optimizations in patches 0005/0006 (in retrospect I might have simply moved 0005+0006 right after 0001, but the patch evolved differently - shouldn't matter here) Now, let's talk about the patches one by one ... PrefetchBuffer + preadv2 (0002+0003) ------------------------------------ After I posted the patch in October, I happened to have an off-list discussion with Andres, and he suggested to try ditching the local cache of recently prefetched blocks, and instead: 1) call PrefetchBuffer (which checks if the page is in shared buffers, and skips the prefetch if it's already there) 2) if the page is not in shared buffers, use preadv2(RWF_NOWAIT) to check if it's in the kernel page cache Doing (1) is trivial - PrefetchBuffer() already does the shared buffer check, so 0002 simply removes the custom cache code. Doing (2) needs a bit more code to actually call preadv2() - 0003 adds FileCached() to fd.c, smgrcached() to smgr.c, and then calls it from PrefetchBuffer() right before smgrprefetch(). There's a couple loose ends (e.g. configure should check if preadv2 is supported), but in principle I think this is generally correct. Unfortunately, these changes led to a bunch of clear regressions :-( Take a look at the attached point-4-regressions-small.png, which is page 5 from the full results PDF [1][2]. As before, I plotted this as a huge pivot table with various parameters (test, dataset, prefetch, ...) on the left, and (build, nmatches) on the top. So each column shows timings for a particular patch and query returning nmatches rows. After the pivot table (on the right) is a heatmap, comparing timings for each build to master (the first couple of columns). As usual, the numbers are "timing compared to master" so e.g. 50% means the query completed in 1/2 the time compared to master. Color coding is simple too, green means "good" (speedup), red means "bad" (regression). The higher the saturation, the bigger the difference. I find this visualization handy as it quickly highlights differences between the various patches. Just look for changes in red/green areas. In the points-5-regressions-small.png image, you can see three areas of clear regressions, either compared to the master or the 20231016 patch. All of this is for "uncached" runs, i.e. after instance got restarted and the page cache was dropped too. The first regression is for bitmapscan. The first two builds show no difference compared to master - which makes sense, because the 20231016 patch does not touch any code used by bitmapscan, and the 0003 patch simply uses PrefetchBuffer as is. But then 0004 adds preadv2 to it, and the performance immediately sinks, with timings being ~5-6x higher for queries matching 1k-100k rows. The patches 0005/0006 can't possibly improve this, because visibilitymap are entirely unrelated to bitmapscans, and so is the small LRU to detect sequential patterns. The indexscan regression #1 shows a similar pattern, but in the opposite direction - indesxcan cases massively improved with the 20231016 patch (and even after just using PrefetchBuffer) revert back to master with 0003 (adding preadv2). Ditching the preadv2 restores the gains (the last build results are nicely green again). The indexscan regression #2 is interesting too, and it illustrates the importance of detecting sequential access patterns. It shows that as soon as we call PrefetBuffer() directly, the timings increase to maybe 2-5x compared to master. That's pretty terrible. Once the small LRU cache used to detect sequential patterns is added back, the performance recovers and regression disappears. Clearly, this detection matters. Unfortunately, the LRU can't do anything for the two other regresisons, because those are on random/cyclic patterns, so the LRU won't work (certainly not for the random case). preadv2 issues? --------------- I'm not entirely sure if I'm using preadv2 somehow wrong, but it doesn't seem to perform terribly well in this use case. I decided to do some microbenchmarks, measuring how long it takes to do preadv2 when the pages are [not] in cache etc. The C files are at [3]. preadv2-test simply reads file twice, first with NOWAIT and then without it. With clean page cache, the results look like this: file: ./tmp.img size: 1073741824 (131072) block 8192 check 8192 preadv2 NOWAIT time 78472 us calls 131072 hits 0 misses 131072 preadv2 WAIT time 9849082 us calls 131072 hits 131072 misses 0 and then, if you run it again with the file still being in page cache: file: ./tmp.img size: 1073741824 (131072) block 8192 check 8192 preadv2 NOWAIT time 258880 us calls 131072 hits 131072 misses 0 preadv2 WAIT time 213196 us calls 131072 hits 131072 misses 0 This is pretty terrible, IMO. It says that if the page is not in cache, the preadv2 calls take ~80ms. Which is very cheap, compared to the total read time (so if we can speed that up by prefetching, it's worth it). But if the file is already in cache, it takes ~260ms, and actually exceeds the time needed to just do preadv2() without the NOWAIT flag. AFAICS the problem is preadv2() doesn't just check if the data is available, it also copies the data and all that. But even if we only ask for the first byte, it's still way more expensive than with empty cache: file: ./tmp.img size: 1073741824 (131072) block 8192 check 1 preadv2 NOWAIT time 119751 us calls 131072 hits 131072 misses 0 preadv2 WAIT time 208136 us calls 131072 hits 131072 misses 0 There's also a fadvise-test microbenchmark that just does fadvise all the time, and even that is way cheaper than using preadv2(NOWAIT) in both cases: no cache: file: ./tmp.img size: 1073741824 (131072) block 8192 fadvise time 631686 us calls 131072 hits 0 misses 0 preadv2 time 207483 us calls 131072 hits 131072 misses 0 cache: file: ./tmp.img size: 1073741824 (131072) block 8192 fadvise time 79874 us calls 131072 hits 0 misses 0 preadv2 time 239141 us calls 131072 hits 131072 misses 0 So that's 300ms vs. 500ms in the caches case (the difference in the no-cache case is even more significant). It's entirely possible I'm doing something wrong, or maybe I just think about this the wrong way, but I can't quite imagine this being useful for this working - at least not for reasonably good local storage. Maybe it could help for slow/remote storage, or something? For now, I think the right approach is to go back to the cache of recently prefetched blocks. I liked on the preadv2 approach is that it knows exactly what is currently in page cache, while the local cache is just an approximation cache of recently prefetched blocks. And it also knows about stuff prefetched by other backends, while the local cache is private to the particular backend (or even to the particular scan node). But the local cache seems to perform much better, so there's that. LRU cache of recent blocks (0004) --------------------------------- The importance of this optimization is clearly visible in the regression image mentioned earlier - the "indexscan regression #2" shows that the sequential pattern regresses with 0002+0003 patches, but once the small LRU cache is introduced back and uses to skip prefetching for sequential patterns, the regression disappears. Ofc, this is part of the origina 20231016 patch, so going back to that version naturally includes this. visibility map optimizations (0005/0006) ---------------------------------------- Earlier benchmark results showed a bit annoying regression for index-only scans that don't need prefetching (i.e. with all pages all-visible). There was quite a bit of inefficiency because both the prefetcher and IOS code accessed the visibilitymap independently, and the prefetcher did that in a rather inefficient way. These patches make the prefetcher more efficient by reusing buffer, and also share the visibility info between prefetcher and the IOS code. I'm sure this needs more work / cleanup, but the regresion is mostly gone, as illustrated by the attached point-0-ios-improvement-small.png. layering questions ------------------ Aside from the preadv2() question, the main open question remains to be the "layering", i.e. which code should be responsible for prefetching. At the moment all the magic happens in indexam.c, in index_getnext_* functions, so that all callers benefit from prefetching. But as mentioned earlier in this thread, indexam.c seems to be the wrong layer, and I think I agree. The problem is - the prefetching needs to happen in index_getnext_* so that all index_getnext_* callers benefit from it. We could do that in the executor for index_getnext_tid(), but that's a bit weird - it'd work for index-only scans, but the primary target is regular index scans, which calls index_getnext_slot(). However, it seems it'd be good if the prefetcher and the executor code could exchange/share information more easily. Take for example the visibilitymap stuff in IOS in patches 0005/0006). I made it work, but it sure looks inconvenient, partially due to the split between executor and indexam code. The only idea I have is to have the prefetcher code somewhere in the executor, but then pass it to index_getnext_* functions, either as a new parameter (with NULL => no prefetching), or maybe as a field of scandesc (but that seems wrong, to point from the desc to something that's essentially a part of the executor state). There's also the thing that the prefetcher is part of IndexScanDesc, but it really should be in the IndexScanState. That's weird, but mostly down to my general laziness. regards [1] https://github.com/tvondra/index-prefetch-tests/blob/master/2023-11-23/pdf/point.pdf [2] https://github.com/tvondra/index-prefetch-tests/blob/master/2023-11-23/png/point-4.png [3] https://github.com/tvondra/index-prefetch-tests/tree/master/2023-11-23/preadv-tests -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company Attachments: [text/x-patch] v20231124-0001-prefetch-2023-10-16.patch (53.5K, 2-v20231124-0001-prefetch-2023-10-16.patch) download | inline diff: From 3bf9b6c3554b5aa70c189bf4d04c9f063a8ac49b Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Fri, 17 Nov 2023 23:54:19 +0100 Subject: [PATCH v20231124 1/7] prefetch 2023-10-16 Patch version shared on 2023/10/16, with only minor tweaks in comments. https://www.postgresql.org/message-id/06bb7d02-2c44-3062-731e-a735ba13da7e%40enterprisedb.com --- src/backend/access/heap/heapam_handler.c | 12 +- src/backend/access/index/genam.c | 31 +- src/backend/access/index/indexam.c | 659 ++++++++++++++++++++++- src/backend/commands/explain.c | 18 + src/backend/executor/execIndexing.c | 6 +- src/backend/executor/execReplication.c | 9 +- src/backend/executor/instrument.c | 4 + src/backend/executor/nodeIndexonlyscan.c | 73 ++- src/backend/executor/nodeIndexscan.c | 80 ++- src/backend/utils/adt/selfuncs.c | 3 +- src/include/access/genam.h | 101 +++- src/include/access/relscan.h | 9 + src/include/executor/instrument.h | 2 + 13 files changed, 975 insertions(+), 32 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 7c28dafb728..89474078951 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -44,6 +44,7 @@ #include "storage/smgr.h" #include "utils/builtins.h" #include "utils/rel.h" +#include "utils/spccache.h" static void reform_and_rewrite_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, @@ -747,6 +748,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, PROGRESS_CLUSTER_INDEX_RELID }; int64 ci_val[2]; + int prefetch_max; + + /* + * Get the prefetch target for the old tablespace (which is what we'll + * read using the index). We'll use it as a reset value too, although + * there should be no rescans for CLUSTER etc. + */ + prefetch_max = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; @@ -755,7 +764,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tableScan = NULL; heapScan = NULL; - indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0); + indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0, + prefetch_max); index_rescan(indexScan, NULL, 0, NULL, 0); } else diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 4ca12006843..d45a209ee3a 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -126,6 +126,9 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys) scan->xs_hitup = NULL; scan->xs_hitupdesc = NULL; + /* Information used for asynchronous prefetching during index scans. */ + scan->xs_prefetch = NULL; + return scan; } @@ -440,8 +443,20 @@ systable_beginscan(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* + * We don't do any prefetching on system catalogs, for two main reasons. + * + * Firstly, we usually do PK lookups, which makes prefetching pointles, + * or we often don't know how many rows to expect (and the numbers tend + * to be fairly low). So it's not clear it'd help. Furthermore, places + * that are sensitive tend to use syscache anyway. + * + * Secondly, we can't call get_tablespace_io_concurrency() because that + * does a sysscan internally, so it might lead to a cycle. We could use + * use effective_io_concurrency, but it doesn't seem worth it. + */ sysscan->iscan = index_beginscan(heapRelation, irel, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; } @@ -696,8 +711,20 @@ systable_beginscan_ordered(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* + * We don't do any prefetching on system catalogs, for two main reasons. + * + * Firstly, we usually do PK lookups, which makes prefetching pointles, + * or we often don't know how many rows to expect (and the numbers tend + * to be fairly low). So it's not clear it'd help. Furthermore, places + * that are sensitive tend to use syscache anyway. + * + * Secondly, we can't call get_tablespace_io_concurrency() because that + * does a sysscan internally, so it might lead to a cycle. We could use + * use effective_io_concurrency, but it doesn't seem worth it. + */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index b25b03f7abc..51feece527a 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -49,16 +49,19 @@ #include "access/relscan.h" #include "access/tableam.h" #include "access/transam.h" +#include "access/visibilitymap.h" #include "access/xlog.h" #include "catalog/index.h" #include "catalog/pg_amproc.h" #include "catalog/pg_type.h" #include "commands/defrem.h" +#include "common/hashfn.h" #include "nodes/makefuncs.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" #include "storage/predicate.h" +#include "utils/lsyscache.h" #include "utils/ruleutils.h" #include "utils/snapmgr.h" #include "utils/syscache.h" @@ -106,7 +109,10 @@ do { \ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap); + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_max); + +static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible); /* ---------------------------------------------------------------- @@ -200,18 +206,42 @@ index_insert(Relation indexRelation, * index_beginscan - start a scan of an index with amgettuple * * Caller must be holding suitable locks on the heap and the index. + * + * prefetch_max determines if prefetching is requested for this index scan, + * and how far ahead we want to prefetch + * + * Setting prefetch_max to 0 disables prefetching for the index scan. We do + * this for two reasons - for scans on system catalogs, and/or for cases where + * prefetching is expected to be pointless (like IOS). + * + * For system catalogs, we usually either scan by a PK value, or we we expect + * only few rows (or rather we don't know how many rows to expect). Also, we + * need to prevent infinite in the get_tablespace_io_concurrency() call - it + * does an index scan internally. So we simply disable prefetching for system + * catalogs. We could deal with this by picking a conservative static target + * (e.g. effective_io_concurrency, capped to something), but places that are + * performance sensitive likely use syscache anyway, and catalogs tend to be + * very small and hot. So we don't bother. + * + * For IOS, we expect to not need most heap pages (that's the whole point of + * IOS, actually), and prefetching them might lead to a lot of wasted I/O. + * + * XXX Not sure the infinite loop can still happen, now that the target lookup + * moved to callers of index_beginscan. */ IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys) + int nkeys, int norderbys, + int prefetch_max) { IndexScanDesc scan; Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false); + scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, + NULL, false, prefetch_max); /* * Save additional parameters into the scandesc. Everything else was set @@ -241,7 +271,8 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false); + /* No prefetch in bitmap scans, prefetch is done by the heap scan. */ + scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, 0); /* * Save additional parameters into the scandesc. Everything else was set @@ -258,7 +289,8 @@ index_beginscan_bitmap(Relation indexRelation, static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap) + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_max) { IndexScanDesc scan; @@ -282,6 +314,29 @@ index_beginscan_internal(Relation indexRelation, scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; + /* + * With prefetching requested, initialize the prefetcher state. + * + * FIXME This should really be in the IndexScanState, not IndexScanDesc + * (certainly the queues etc). But index_getnext_tid only gets the scan + * descriptor, so how else would we pass it? Seems like a sign of wrong + * layer doing the prefetching. + */ + if ((prefetch_max > 0) && + (io_direct_flags & IO_DIRECT_DATA) == 0) /* no prefetching for direct I/O */ + { + IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); + + prefetcher->queueIndex = 0; + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + + prefetcher->prefetchTarget = 0; + prefetcher->prefetchMaxTarget = prefetch_max; + + scan->xs_prefetch = prefetcher; + } + return scan; } @@ -317,6 +372,20 @@ index_rescan(IndexScanDesc scan, scan->indexRelation->rd_indam->amrescan(scan, keys, nkeys, orderbys, norderbys); + + /* If we're prefetching for this index, maybe reset some of the state. */ + if (scan->xs_prefetch != NULL) + { + IndexPrefetch prefetcher = scan->xs_prefetch; + + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + prefetcher->queueIndex = 0; + prefetcher->prefetchDone = false; + + /* restart the incremental ramp-up */ + prefetcher->prefetchTarget = 0; + } } /* ---------------- @@ -345,6 +414,23 @@ index_endscan(IndexScanDesc scan) if (scan->xs_temp_snap) UnregisterSnapshot(scan->xs_snapshot); + /* + * If prefetching was enabled for this scan, log prefetch stats. + * + * FIXME This should really go to EXPLAIN ANALYZE instead. + */ + if (scan->xs_prefetch) + { + IndexPrefetch prefetch = scan->xs_prefetch; + + elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, + prefetch->countAll, + prefetch->countPrefetch, + prefetch->countPrefetch * 100.0 / prefetch->countAll, + prefetch->countSkipCached, + prefetch->countSkipSequential); + } + /* Release the scan data structure itself */ IndexScanEnd(scan); } @@ -490,7 +576,8 @@ index_parallelrescan(IndexScanDesc scan) */ IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, - int norderbys, ParallelIndexScanDesc pscan) + int norderbys, ParallelIndexScanDesc pscan, + int prefetch_max) { Snapshot snapshot; IndexScanDesc scan; @@ -499,7 +586,7 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, snapshot = RestoreSnapshot(pscan->ps_snapshot_data); RegisterSnapshot(snapshot); scan = index_beginscan_internal(indexrel, nkeys, norderbys, snapshot, - pscan, true); + pscan, true, prefetch_max); /* * Save additional parameters into the scandesc. Everything else was set @@ -623,20 +710,95 @@ index_fetch_heap(IndexScanDesc scan, TupleTableSlot *slot) bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot) { + IndexPrefetch prefetch = scan->xs_prefetch; /* for convenience */ + for (;;) { + /* + * If the prefetching is still active (i.e. enabled and we still + * haven't finished reading TIDs from the scan), read enough TIDs into + * the queue until we hit the current target. + */ + if (PREFETCH_ACTIVE(prefetch)) + { + /* + * Ramp up the prefetch distance incrementally. + * + * Intentionally done as first, before reading the TIDs into the + * queue, so that there's always at least one item. Otherwise we + * might get into a situation where we start with target=0 and no + * TIDs loaded. + */ + prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, + prefetch->prefetchMaxTarget); + + /* + * Now read TIDs from the index until the queue is full (with + * respect to the current prefetch target). + */ + while (!PREFETCH_FULL(prefetch)) + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* + * If we're out of index entries, we're done (and we mark the + * the prefetcher as inactive). + */ + if (tid == NULL) + { + prefetch->prefetchDone = true; + break; + } + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; + prefetch->queueEnd++; + + /* + * Issue the actuall prefetch requests for the new TID. + * + * FIXME For IOS, this should prefetch only pages that are not + * fully visible. + */ + index_prefetch(scan, tid, false); + } + } + if (!scan->xs_heap_continue) { - ItemPointer tid; + /* + * With prefetching enabled (even if we already finished reading + * all TIDs from the index scan), we need to return a TID from the + * queue. Otherwise, we just get the next TID from the scan + * directly. + */ + if (PREFETCH_ENABLED(prefetch)) + { + /* Did we reach the end of the scan and the queue is empty? */ + if (PREFETCH_DONE(prefetch)) + break; - /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; + prefetch->queueIndex++; + } + else /* not prefetching, just do the regular work */ + { + ItemPointer tid; - /* If we're out of index entries, we're done */ - if (tid == NULL) - break; + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* If we're out of index entries, we're done */ + if (tid == NULL) + break; + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + } - Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); } /* @@ -988,3 +1150,472 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, return build_local_reloptions(&relopts, attoptions, validate); } + +/* + * index_prefetch_is_sequential + * Track the block number and check if the I/O pattern is sequential, + * or if the same block was just prefetched. + * + * Prefetching is cheap, but for some access patterns the benefits are small + * compared to the extra overhead. In particular, for sequential access the + * read-ahead performed by the OS is very effective/efficient. Doing more + * prefetching is just increasing the costs. + * + * This tries to identify simple sequential patterns, so that we can skip + * the prefetching request. This is implemented by having a small queue + * of block numbers, and checking it before prefetching another block. + * + * We look at the preceding PREFETCH_SEQ_PATTERN_BLOCKS blocks, and see if + * they are sequential. We also check if the block is the same as the last + * request (which is not sequential). + * + * Note that the main prefetch queue is not really useful for this, as it + * stores TIDs while we care about block numbers. Consider a sorted table, + * with a perfectly sequential pattern when accessed through an index. Each + * heap page may have dozens of TIDs, but we need to check block numbers. + * We could keep enough TIDs to cover enough blocks, but then we also need + * to walk those when checking the pattern (in hot path). + * + * So instead, we maintain a small separate queue of block numbers, and we use + * this instead. + * + * Returns true if the block is in a sequential pattern (and so should not be + * prefetched), or false (not sequential, should be prefetched). + * + * XXX The name is a bit misleading, as it also adds the block number to the + * block queue and checks if the block is the same as the last one (which + * does not require a sequential pattern). + */ +static bool +index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) +{ + int idx; + + /* + * If the block queue is empty, just store the block and we're done (it's + * neither a sequential pattern, neither recently prefetched block). + */ + if (prefetch->blockIndex == 0) + { + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + return false; + } + + /* + * Check if it's the same as the immediately preceding block. We don't + * want to prefetch the same block over and over (which would happen for + * well correlated indexes). + * + * In principle we could rely on index_prefetch_add_cache doing this using + * the full cache, but this check is much cheaper and we need to look at + * the preceding block anyway, so we just do it. + * + * XXX Notice we haven't added the block to the block queue yet, and there + * is a preceding block (i.e. blockIndex-1 is valid). + */ + if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) + return true; + + /* + * Add the block number to the queue. + * + * We do this before checking if the pattern, because we want to know + * about the block even if we end up skipping the prefetch. Otherwise we'd + * not be able to detect longer sequential pattens - we'd skip one block + * but then fail to skip the next couple blocks even in a perfect + * sequential pattern. This ocillation might even prevent the OS + * read-ahead from kicking in. + */ + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + + /* + * Check if the last couple blocks are in a sequential pattern. We look + * for a sequential pattern of PREFETCH_SEQ_PATTERN_BLOCKS (4 by default), + * so we look for patterns of 5 pages (40kB) including the new block. + * + * XXX Perhaps this should be tied to effective_io_concurrency somehow? + * + * XXX Could it be harmful that we read the queue backwards? Maybe memory + * prefetching works better for the forward direction? + */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) + { + /* + * Are there enough requests to confirm a sequential pattern? We only + * consider something to be sequential after finding a sequence of + * PREFETCH_SEQ_PATTERN_BLOCKS blocks. + * + * FIXME Better to move this outside the loop. + */ + if (prefetch->blockIndex < i) + return false; + + /* + * Calculate index of the earlier block (we need to do -1 as we + * already incremented the index when adding the new block to the + * queue). + */ + idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); + + /* + * For a sequential pattern, blocks "k" step ago needs to have block + * number by "k" smaller compared to the current block. + */ + if (prefetch->blockItems[idx] != (block - i)) + return false; + } + + return true; +} + +/* + * index_prefetch_add_cache + * Add a block to the cache, check if it was recently prefetched. + * + * We don't want to prefetch blocks that we already prefetched recently. It's + * cheap but not free, and the overhead may have measurable impact. + * + * This check needs to be very cheap, even with fairly large caches (hundreds + * of entries, see PREFETCH_CACHE_SIZE). + * + * A simple queue would allow expiring the requests, but checking if it + * contains a particular block prefetched would be expensive (linear search). + * Another option would be a simple hash table, which has fast lookup but + * does not allow expiring entries cheaply. + * + * The cache does not need to be perfect, we can accept false + * positives/negatives, as long as the rate is reasonably low. We also need + * to expire entries, so that only "recent" requests are remembered. + * + * We use a hybrid cache that is organized as many small LRU caches. Each + * block is mapped to a particular LRU by hashing (so it's a bit like a + * hash table). The LRU caches are tiny (e.g. 8 entries), and the expiration + * happens at the level of a single LRU (by tracking only the 8 most recent requests). + * + * This allows quick searches and expiration, but with false negatives (when a + * particular LRU has too many collisions, we may evict entries that are more + * recent than some other LRU). + * + * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 + * prefetch request in total (these are the default parameters.) + * + * The recency is determined using a prefetch counter, incremented every + * time we end up prefetching a block. The counter is uint64, so it should + * not wrap (125 zebibytes, would take ~4 million years at 1GB/s). + * + * To check if a block was prefetched recently, we calculate hash(block), + * and then linearly search if the tiny LRU has entry for the same block + * and request less than PREFETCH_CACHE_SIZE ago. + * + * At the same time, we either update the entry (for the queried block) if + * found, or replace the oldest/empty entry. + * + * If the block was not recently prefetched (i.e. we want to prefetch it), + * we increment the counter. + * + * Returns true if the block was recently prefetched (and thus we don't + * need to prefetch it again), or false (should do a prefetch). + * + * XXX It's a bit confusing these return values are inverse compared to + * what index_prefetch_is_sequential does. + */ +static bool +index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) +{ + PrefetchCacheEntry *entry; + + /* map the block number the the LRU */ + int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; + + /* age/index of the oldest entry in the LRU, to maybe use */ + uint64 oldestRequest = PG_UINT64_MAX; + int oldestIndex = -1; + + /* + * First add the block to the (tiny) top-level LRU cache and see if it's + * part of a sequential pattern. In this case we just ignore the block and + * don't prefetch it - we expect read-ahead to do a better job. + * + * XXX Maybe we should still add the block to the hybrid cache, in case we + * happen to access it later? That might help if we first scan a lot of + * the table sequentially, and then randomly. Not sure that's very likely + * with index access, though. + */ + if (index_prefetch_is_sequential(prefetch, block)) + { + prefetch->countSkipSequential++; + return true; + } + + /* + * See if we recently prefetched this block - we simply scan the LRU + * linearly. While doing that, we also track the oldest entry, so that we + * know where to put the block if we don't find a matching entry. + */ + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) + { + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; + + /* Is this the oldest prefetch request in this LRU? */ + if (entry->request < oldestRequest) + { + oldestRequest = entry->request; + oldestIndex = i; + } + + /* + * If the entry is unused (identified by request being set to 0), + * we're done. Notice the field is uint64, so empty entry is + * guaranteed to be the oldest one. + */ + if (entry->request == 0) + continue; + + /* Is this entry for the same block as the current request? */ + if (entry->block == block) + { + bool prefetched; + + /* + * Is the old request sufficiently recent? If yes, we treat the + * block as already prefetched. + * + * XXX We do add the cache size to the request in order not to + * have issues with uint64 underflows. + */ + prefetched = ((entry->request + PREFETCH_CACHE_SIZE) >= prefetch->prefetchReqNumber); + + /* Update the request number. */ + entry->request = ++prefetch->prefetchReqNumber; + + prefetch->countSkipCached += (prefetched) ? 1 : 0; + + return prefetched; + } + } + + /* + * We didn't find the block in the LRU, so store it either in an empty + * entry, or in the "oldest" prefetch request in this LRU. + */ + Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); + + /* FIXME do a nice macro */ + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; + + entry->block = block; + entry->request = ++prefetch->prefetchReqNumber; + + /* not in the prefetch cache */ + return false; +} + +/* + * index_prefetch + * Prefetch the TID, unless it's sequential or recently prefetched. + * + * XXX Some ideas how to auto-tune the prefetching, so that unnecessary + * prefetching does not cause significant regressions (e.g. for nestloop + * with inner index scan). We could track number of rescans and number of + * items (TIDs) actually returned from the scan. Then we could calculate + * rows / rescan and use that to clamp prefetch target. + * + * That'd help with cases when a scan matches only very few rows, far less + * than the prefetchTarget, because the unnecessary prefetches are wasted + * I/O. Imagine a LIMIT on top of index scan, or something like that. + * + * Another option is to use the planner estimates - we know how many rows we're + * expecting to fetch (on average, assuming the estimates are reasonably + * accurate), so why not to use that? + * + * Of course, we could/should combine these two approaches. + * + * XXX The prefetching may interfere with the patch allowing us to evaluate + * conditions on the index tuple, in which case we may not need the heap + * tuple. Maybe if there's such filter, we should prefetch only pages that + * are not all-visible (and the same idea would also work for IOS), but + * it also makes the indexing a bit "aware" of the visibility stuff (which + * seems a somewhat wrong). Also, maybe we should consider the filter selectivity + * (if the index-only filter is expected to eliminate only few rows, then + * the vm check is pointless). Maybe this could/should be auto-tuning too, + * i.e. we could track how many heap tuples were needed after all, and then + * we would consider this when deciding whether to prefetch all-visible + * pages or not (matters only for regular index scans, not IOS). + * + * XXX Maybe we could/should also prefetch the next index block, e.g. stored + * in BTScanPosData.nextPage. + */ +static void +index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) +{ + IndexPrefetch prefetch = scan->xs_prefetch; + BlockNumber block; + + /* + * No heap relation means bitmap index scan, which does prefetching at the + * bitmap heap scan, so no prefetch here (we can't do it anyway, without + * the heap) + * + * XXX But in this case we should have prefetchMaxTarget=0, because in + * index_bebinscan_bitmap() we disable prefetching. So maybe we should + * just check that. + */ + if (!prefetch) + return; + + /* + * If we got here, prefetching is enabled and it's a node that supports + * prefetching (i.e. it can't be a bitmap index scan). + */ + Assert(scan->heapRelation); + + block = ItemPointerGetBlockNumber(tid); + + /* + * When prefetching for IOS, we want to only prefetch pages that are not + * marked as all-visible (because not fetching all-visible pages is the + * point of IOS). + * + * XXX This is not great, because it releases the VM buffer for each TID + * we consider to prefetch. We should reuse that somehow, similar to the + * actual IOS code. Ideally, we should use the same ioss_VMBuffer (if + * we can propagate it here). Or at least do it for a bulk of prefetches, + * although that's not very useful - after the ramp-up we will prefetch + * the pages one by one anyway. + */ + if (skip_all_visible) + { + bool all_visible; + Buffer vmbuffer = InvalidBuffer; + + all_visible = VM_ALL_VISIBLE(scan->heapRelation, + block, + &vmbuffer); + + if (vmbuffer != InvalidBuffer) + ReleaseBuffer(vmbuffer); + + if (all_visible) + return; + } + + /* + * Do not prefetch the same block over and over again, + * + * This happens e.g. for clustered or naturally correlated indexes (fkey + * to a sequence ID). It's not expensive (the block is in page cache + * already, so no I/O), but it's not free either. + */ + if (!index_prefetch_add_cache(prefetch, block)) + { + prefetch->countPrefetch++; + + PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); + pgBufferUsage.blks_prefetches++; + } + + prefetch->countAll++; +} + +/* ---------------- + * index_getnext_tid_prefetch - get the next TID from a scan + * + * The result is the next TID satisfying the scan keys, + * or NULL if no more matching tuples exist. + * + * FIXME not sure this handles xs_heapfetch correctly. + * ---------------- + */ +ItemPointer +index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) +{ + IndexPrefetch prefetch = scan->xs_prefetch; /* for convenience */ + + /* + * If the prefetching is still active (i.e. enabled and we still + * haven't finished reading TIDs from the scan), read enough TIDs into + * the queue until we hit the current target. + */ + if (PREFETCH_ACTIVE(prefetch)) + { + /* + * Ramp up the prefetch distance incrementally. + * + * Intentionally done as first, before reading the TIDs into the + * queue, so that there's always at least one item. Otherwise we + * might get into a situation where we start with target=0 and no + * TIDs loaded. + */ + prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, + prefetch->prefetchMaxTarget); + + /* + * Now read TIDs from the index until the queue is full (with + * respect to the current prefetch target). + */ + while (!PREFETCH_FULL(prefetch)) + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* + * If we're out of index entries, we're done (and we mark the + * the prefetcher as inactive). + */ + if (tid == NULL) + { + prefetch->prefetchDone = true; + break; + } + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; + prefetch->queueEnd++; + + /* + * Issue the actuall prefetch requests for the new TID. + * + * XXX index_getnext_tid_prefetch is only called for IOS (for now), + * so skip prefetching of all-visible pages. + */ + index_prefetch(scan, tid, true); + } + } + + /* + * With prefetching enabled (even if we already finished reading + * all TIDs from the index scan), we need to return a TID from the + * queue. Otherwise, we just get the next TID from the scan + * directly. + */ + if (PREFETCH_ENABLED(prefetch)) + { + /* Did we reach the end of the scan and the queue is empty? */ + if (PREFETCH_DONE(prefetch)) + return NULL; + + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; + prefetch->queueIndex++; + } + else /* not prefetching, just do the regular work */ + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid(scan, direction); + + /* If we're out of index entries, we're done */ + if (tid == NULL) + return NULL; + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + } + + /* Return the TID of the tuple we found. */ + return &scan->xs_heaptid; +} diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index f1d71bc54e8..6810996edfd 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3568,6 +3568,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) !INSTR_TIME_IS_ZERO(usage->local_blk_write_time)); bool has_temp_timing = (!INSTR_TIME_IS_ZERO(usage->temp_blk_read_time) || !INSTR_TIME_IS_ZERO(usage->temp_blk_write_time)); + bool has_prefetches = (usage->blks_prefetches > 0); bool show_planning = (planning && (has_shared || has_local || has_temp || has_shared_timing || @@ -3679,6 +3680,23 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) appendStringInfoChar(es->str, '\n'); } + /* As above, show only positive counter values. */ + if (has_prefetches) + { + ExplainIndentText(es); + appendStringInfoString(es->str, "Prefetches:"); + + if (usage->blks_prefetches > 0) + appendStringInfo(es->str, " blocks=%lld", + (long long) usage->blks_prefetches); + + if (usage->blks_prefetch_rounds > 0) + appendStringInfo(es->str, " rounds=%lld", + (long long) usage->blks_prefetch_rounds); + + appendStringInfoChar(es->str, '\n'); + } + if (show_planning) es->indent--; } diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 384b39839a0..70a7b65323e 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -765,11 +765,15 @@ check_exclusion_or_unique_constraint(Relation heap, Relation index, /* * May have to restart scan from this point if a potential conflict is * found. + * + * XXX Should this do index prefetch? Probably not worth it for unique + * constraints, I guess? Otherwise we should calculate prefetch_target + * just like in nodeIndexscan etc. */ retry: conflict = false; found_self = false; - index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0); + index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0); index_rescan(index_scan, scankeys, indnkeyatts, NULL, 0); while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot)) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 81f27042bc4..91676ccff95 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -204,8 +204,13 @@ RelationFindReplTupleByIndex(Relation rel, Oid idxoid, /* Build scan key. */ skey_attoff = build_replindex_scan_key(skey, rel, idxrel, searchslot); - /* Start an index scan. */ - scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0); + /* Start an index scan. + * + * XXX Should this do index prefetching? We're looking for a single tuple, + * probably using a PK / UNIQUE index, so does not seem worth it. If we + * reconsider this, calclate prefetch_target like in nodeIndexscan. + */ + scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0); retry: found = false; diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c index c383f34c066..0011d9f679c 100644 --- a/src/backend/executor/instrument.c +++ b/src/backend/executor/instrument.c @@ -235,6 +235,8 @@ BufferUsageAdd(BufferUsage *dst, const BufferUsage *add) dst->local_blks_written += add->local_blks_written; dst->temp_blks_read += add->temp_blks_read; dst->temp_blks_written += add->temp_blks_written; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds; + dst->blks_prefetches += add->blks_prefetches; INSTR_TIME_ADD(dst->shared_blk_read_time, add->shared_blk_read_time); INSTR_TIME_ADD(dst->shared_blk_write_time, add->shared_blk_write_time); INSTR_TIME_ADD(dst->local_blk_read_time, add->local_blk_read_time); @@ -259,6 +261,8 @@ BufferUsageAccumDiff(BufferUsage *dst, dst->local_blks_written += add->local_blks_written - sub->local_blks_written; dst->temp_blks_read += add->temp_blks_read - sub->temp_blks_read; dst->temp_blks_written += add->temp_blks_written - sub->temp_blks_written; + dst->blks_prefetches += add->blks_prefetches - sub->blks_prefetches; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds - sub->blks_prefetch_rounds; INSTR_TIME_ACCUM_DIFF(dst->shared_blk_read_time, add->shared_blk_read_time, sub->shared_blk_read_time); INSTR_TIME_ACCUM_DIFF(dst->shared_blk_write_time, diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index f1db35665c8..855afd5ba76 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -43,7 +43,7 @@ #include "storage/predicate.h" #include "utils/memutils.h" #include "utils/rel.h" - +#include "utils/spccache.h" static TupleTableSlot *IndexOnlyNext(IndexOnlyScanState *node); static void StoreIndexTuple(TupleTableSlot *slot, IndexTuple itup, @@ -65,6 +65,7 @@ IndexOnlyNext(IndexOnlyScanState *node) IndexScanDesc scandesc; TupleTableSlot *slot; ItemPointer tid; + Relation heapRel = node->ss.ss_currentRelation; /* * extract necessary information from index scan node @@ -83,16 +84,41 @@ IndexOnlyNext(IndexOnlyScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index only scan is not parallel, or if we're * serially executing an index only scan that was planned to be * parallel. + * + * XXX Maybe we should enable prefetching, but prefetch only pages that + * are not all-visible (but checking that from the index code seems like + * a violation of layering etc). + * + * XXX This might lead to IOS being slower than plain index scan, if the + * table has a lot of pages that need recheck. */ scandesc = index_beginscan(node->ss.ss_currentRelation, node->ioss_RelationDesc, estate->es_snapshot, node->ioss_NumScanKeys, - node->ioss_NumOrderByKeys); + node->ioss_NumOrderByKeys, + prefetch_max); node->ioss_ScanDesc = scandesc; @@ -116,7 +142,7 @@ IndexOnlyNext(IndexOnlyScanState *node) /* * OK, now that we have what we need, fetch the next tuple. */ - while ((tid = index_getnext_tid(scandesc, direction)) != NULL) + while ((tid = index_getnext_tid_prefetch(scandesc, direction)) != NULL) { bool tuple_from_heap = false; @@ -646,6 +672,24 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->ioss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -658,7 +702,8 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -696,6 +741,23 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->ioss_ScanDesc = @@ -703,7 +765,8 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 14b9c00217a..a5f5394ef49 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -43,6 +43,7 @@ #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "utils/spccache.h" /* * When an ordering operator is used, tuples fetched from the index that @@ -85,6 +86,7 @@ IndexNext(IndexScanState *node) ScanDirection direction; IndexScanDesc scandesc; TupleTableSlot *slot; + Relation heapRel = node->ss.ss_currentRelation; /* * extract necessary information from index scan node @@ -103,6 +105,21 @@ IndexNext(IndexScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index scan. This + * is essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -111,7 +128,8 @@ IndexNext(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_max); node->iss_ScanDesc = scandesc; @@ -177,6 +195,7 @@ IndexNextWithReorder(IndexScanState *node) Datum *lastfetched_vals; bool *lastfetched_nulls; int cmp; + Relation heapRel = node->ss.ss_currentRelation; estate = node->ss.ps.state; @@ -198,6 +217,21 @@ IndexNextWithReorder(IndexScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -206,7 +240,8 @@ IndexNextWithReorder(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_max); node->iss_ScanDesc = scandesc; @@ -1662,6 +1697,24 @@ ExecIndexScanInitializeDSM(IndexScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -1674,7 +1727,8 @@ ExecIndexScanInitializeDSM(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); /* * If no run-time keys to calculate or they are ready, go ahead and pass @@ -1710,6 +1764,23 @@ ExecIndexScanInitializeWorker(IndexScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->iss_ScanDesc = @@ -1717,7 +1788,8 @@ ExecIndexScanInitializeWorker(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); /* * If no run-time keys to calculate or they are ready, go ahead and pass diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 35c9e3c86fe..9447910f103 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -6284,9 +6284,10 @@ get_actual_variable_endpoint(Relation heapRel, InitNonVacuumableSnapshot(SnapshotNonVacuumable, GlobalVisTestFor(heapRel)); + /* XXX Maybe should do prefetching using the default prefetch parameters? */ index_scan = index_beginscan(heapRel, indexRel, &SnapshotNonVacuumable, - 1, 0); + 1, 0, 0); /* Set it up for index-only scan */ index_scan->xs_want_itup = true; index_rescan(index_scan, scankeys, 1, NULL, 0); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index f31dec6ee0f..e7b915d6ce7 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -17,6 +17,7 @@ #include "access/sdir.h" #include "access/skey.h" #include "nodes/tidbitmap.h" +#include "storage/bufmgr.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -152,7 +153,8 @@ extern bool index_insert(Relation indexRelation, extern IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys); + int nkeys, int norderbys, + int prefetch_max); extern IndexScanDesc index_beginscan_bitmap(Relation indexRelation, Snapshot snapshot, int nkeys); @@ -169,9 +171,12 @@ extern void index_parallelscan_initialize(Relation heapRelation, extern void index_parallelrescan(IndexScanDesc scan); extern IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, - ParallelIndexScanDesc pscan); + ParallelIndexScanDesc pscan, + int prefetch_max); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); +extern ItemPointer index_getnext_tid_prefetch(IndexScanDesc scan, + ScanDirection direction); struct TupleTableSlot; extern bool index_fetch_heap(IndexScanDesc scan, struct TupleTableSlot *slot); extern bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, @@ -230,4 +235,96 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); +/* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. Doesn't need to be perfectly accurate, but we + * aim to make false positives/negatives reasonably low. + */ +typedef struct PrefetchCacheEntry { + BlockNumber block; + uint64 request; +} PrefetchCacheEntry; + +/* + * Size of the cache of recently prefetched blocks - shouldn't be too + * small or too large. 1024 seems about right, it covers ~8MB of data. + * It's somewhat arbitrary, there's no particular formula saying it + * should not be higher/lower. + * + * The cache is structured as an array of small LRU caches, so the total + * size needs to be a multiple of LRU size. The LRU should be tiny to + * keep linear search cheap enough. + * + * XXX Maybe we could consider effective_cache_size or something? + */ +#define PREFETCH_LRU_SIZE 8 +#define PREFETCH_LRU_COUNT 128 +#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) + +/* + * Used to detect sequential patterns (and disable prefetching). + */ +#define PREFETCH_QUEUE_HISTORY 8 +#define PREFETCH_SEQ_PATTERN_BLOCKS 4 + + +typedef struct IndexPrefetchData +{ + /* + * XXX We need to disable this in some cases (e.g. when using index-only + * scans, we don't want to prefetch pages). Or maybe we should prefetch + * only pages that are not all-visible, that'd be even better. + */ + int prefetchTarget; /* how far we should be prefetching */ + int prefetchMaxTarget; /* maximum prefetching distance */ + int prefetchReset; /* reset to this distance on rescan */ + bool prefetchDone; /* did we get all TIDs from the index? */ + + /* runtime statistics */ + uint64 countAll; /* all prefetch requests */ + uint64 countPrefetch; /* actual prefetches */ + uint64 countSkipSequential; + uint64 countSkipCached; + + /* + * Queue of TIDs to prefetch. + * + * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler + * than dynamically adjusting for custom values. + */ + ItemPointerData queueItems[MAX_IO_CONCURRENCY]; + uint64 queueIndex; /* next TID to prefetch */ + uint64 queueStart; /* first valid TID in queue */ + uint64 queueEnd; /* first invalid (empty) TID in queue */ + + /* + * A couple of last prefetched blocks, used to check for certain access + * pattern and skip prefetching - e.g. for sequential access). + * + * XXX Separate from the main queue, because we only want to compare the + * block numbers, not the whole TID. In sequential access it's likely we + * read many items from each page, and we don't want to check many items + * (as that is much more expensive). + */ + BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; + uint64 blockIndex; /* index in the block (points to the first + * empty entry)*/ + + /* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. + */ + uint64 prefetchReqNumber; + PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; + +} IndexPrefetchData; + +#define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) +#define PREFETCH_QUEUE_EMPTY(p) ((p)->queueEnd == (p)->queueIndex) +#define PREFETCH_ENABLED(p) ((p) && ((p)->prefetchMaxTarget > 0)) +#define PREFETCH_FULL(p) ((p)->queueEnd - (p)->queueIndex == (p)->prefetchTarget) +#define PREFETCH_DONE(p) ((p) && ((p)->prefetchDone && PREFETCH_QUEUE_EMPTY(p))) +#define PREFETCH_ACTIVE(p) (PREFETCH_ENABLED(p) && !(p)->prefetchDone) +#define PREFETCH_BLOCK_INDEX(v) ((v) % PREFETCH_QUEUE_HISTORY) + #endif /* GENAM_H */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index d03360eac04..231a30ecc46 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -106,6 +106,12 @@ typedef struct IndexFetchTableData Relation rel; } IndexFetchTableData; +/* + * Forward declarations, defined in genam.h. + */ +typedef struct IndexPrefetchData IndexPrefetchData; +typedef struct IndexPrefetchData *IndexPrefetch; + /* * We use the same IndexScanDescData structure for both amgettuple-based * and amgetbitmap-based index scans. Some fields are only relevant in @@ -162,6 +168,9 @@ typedef struct IndexScanDescData bool *xs_orderbynulls; bool xs_recheckorderby; + /* prefetching state (or NULL if disabled for this scan) */ + IndexPrefetchData *xs_prefetch; + /* parallel index scan information, in shared memory */ struct ParallelIndexScanDescData *parallel_scan; } IndexScanDescData; diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index d5d69941c52..f53fb4a1e51 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -33,6 +33,8 @@ typedef struct BufferUsage int64 local_blks_written; /* # of local disk blocks written */ int64 temp_blks_read; /* # of temp blocks read */ int64 temp_blks_written; /* # of temp blocks written */ + int64 blks_prefetch_rounds; /* # of prefetch rounds */ + int64 blks_prefetches; /* # of buffers prefetched */ instr_time shared_blk_read_time; /* time spent reading shared blocks */ instr_time shared_blk_write_time; /* time spent writing shared blocks */ instr_time local_blk_read_time; /* time spent reading local blocks */ -- 2.42.0 [text/x-patch] v20231124-0002-rely-on-PrefetchBuffer-instead-of-custom-c.patch (22.3K, 3-v20231124-0002-rely-on-PrefetchBuffer-instead-of-custom-c.patch) download | inline diff: From a5a897a6b77b9db99186092060d55b34491acbf2 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Sat, 18 Nov 2023 00:32:33 +0100 Subject: [PATCH v20231124 2/7] rely on PrefetchBuffer instead of custom cache Instead of maintaining a custom cache of recently prefetched blocks, rely on PrefetchBuffer doing the right thing. This only checks shared buffers, though, there's no attempt to determine if block is in page cache. However, it's a shared cache, not restricted to a single process. --- src/backend/access/index/indexam.c | 400 +++-------------------- src/backend/executor/nodeIndexonlyscan.c | 8 +- src/include/access/genam.h | 53 --- src/include/access/relscan.h | 1 + 4 files changed, 52 insertions(+), 410 deletions(-) diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 51feece527a..54a704338f1 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -112,6 +112,8 @@ static IndexScanDesc index_beginscan_internal(Relation indexRelation, ParallelIndexScanDesc pscan, bool temp_snap, int prefetch_max); +static void index_prefetch_tids(IndexScanDesc scan, ScanDirection direction); +static ItemPointer index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction); static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible); @@ -313,6 +315,7 @@ index_beginscan_internal(Relation indexRelation, /* Initialize information for parallel scan. */ scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; + scan->indexonly = false; /* * With prefetching requested, initialize the prefetcher state. @@ -608,8 +611,8 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, * or NULL if no more matching tuples exist. * ---------------- */ -ItemPointer -index_getnext_tid(IndexScanDesc scan, ScanDirection direction) +static ItemPointer +index_getnext_tid_internal(IndexScanDesc scan, ScanDirection direction) { bool found; @@ -710,95 +713,23 @@ index_fetch_heap(IndexScanDesc scan, TupleTableSlot *slot) bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot) { - IndexPrefetch prefetch = scan->xs_prefetch; /* for convenience */ - for (;;) { - /* - * If the prefetching is still active (i.e. enabled and we still - * haven't finished reading TIDs from the scan), read enough TIDs into - * the queue until we hit the current target. - */ - if (PREFETCH_ACTIVE(prefetch)) - { - /* - * Ramp up the prefetch distance incrementally. - * - * Intentionally done as first, before reading the TIDs into the - * queue, so that there's always at least one item. Otherwise we - * might get into a situation where we start with target=0 and no - * TIDs loaded. - */ - prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, - prefetch->prefetchMaxTarget); - - /* - * Now read TIDs from the index until the queue is full (with - * respect to the current prefetch target). - */ - while (!PREFETCH_FULL(prefetch)) - { - ItemPointer tid; - - /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); - - /* - * If we're out of index entries, we're done (and we mark the - * the prefetcher as inactive). - */ - if (tid == NULL) - { - prefetch->prefetchDone = true; - break; - } - - Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); - - prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; - prefetch->queueEnd++; - - /* - * Issue the actuall prefetch requests for the new TID. - * - * FIXME For IOS, this should prefetch only pages that are not - * fully visible. - */ - index_prefetch(scan, tid, false); - } - } + /* Do prefetching (if requested/enabled). */ + index_prefetch_tids(scan, direction); if (!scan->xs_heap_continue) { - /* - * With prefetching enabled (even if we already finished reading - * all TIDs from the index scan), we need to return a TID from the - * queue. Otherwise, we just get the next TID from the scan - * directly. - */ - if (PREFETCH_ENABLED(prefetch)) - { - /* Did we reach the end of the scan and the queue is empty? */ - if (PREFETCH_DONE(prefetch)) - break; - - scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; - prefetch->queueIndex++; - } - else /* not prefetching, just do the regular work */ - { - ItemPointer tid; - - /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + ItemPointer tid; - /* If we're out of index entries, we're done */ - if (tid == NULL) - break; + /* Time to fetch the next TID from the index */ + tid = index_prefetch_get_tid(scan, direction); - Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); - } + /* If we're out of index entries, we're done */ + if (tid == NULL) + break; + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); } /* @@ -1151,267 +1082,6 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, return build_local_reloptions(&relopts, attoptions, validate); } -/* - * index_prefetch_is_sequential - * Track the block number and check if the I/O pattern is sequential, - * or if the same block was just prefetched. - * - * Prefetching is cheap, but for some access patterns the benefits are small - * compared to the extra overhead. In particular, for sequential access the - * read-ahead performed by the OS is very effective/efficient. Doing more - * prefetching is just increasing the costs. - * - * This tries to identify simple sequential patterns, so that we can skip - * the prefetching request. This is implemented by having a small queue - * of block numbers, and checking it before prefetching another block. - * - * We look at the preceding PREFETCH_SEQ_PATTERN_BLOCKS blocks, and see if - * they are sequential. We also check if the block is the same as the last - * request (which is not sequential). - * - * Note that the main prefetch queue is not really useful for this, as it - * stores TIDs while we care about block numbers. Consider a sorted table, - * with a perfectly sequential pattern when accessed through an index. Each - * heap page may have dozens of TIDs, but we need to check block numbers. - * We could keep enough TIDs to cover enough blocks, but then we also need - * to walk those when checking the pattern (in hot path). - * - * So instead, we maintain a small separate queue of block numbers, and we use - * this instead. - * - * Returns true if the block is in a sequential pattern (and so should not be - * prefetched), or false (not sequential, should be prefetched). - * - * XXX The name is a bit misleading, as it also adds the block number to the - * block queue and checks if the block is the same as the last one (which - * does not require a sequential pattern). - */ -static bool -index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) -{ - int idx; - - /* - * If the block queue is empty, just store the block and we're done (it's - * neither a sequential pattern, neither recently prefetched block). - */ - if (prefetch->blockIndex == 0) - { - prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; - prefetch->blockIndex++; - return false; - } - - /* - * Check if it's the same as the immediately preceding block. We don't - * want to prefetch the same block over and over (which would happen for - * well correlated indexes). - * - * In principle we could rely on index_prefetch_add_cache doing this using - * the full cache, but this check is much cheaper and we need to look at - * the preceding block anyway, so we just do it. - * - * XXX Notice we haven't added the block to the block queue yet, and there - * is a preceding block (i.e. blockIndex-1 is valid). - */ - if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) - return true; - - /* - * Add the block number to the queue. - * - * We do this before checking if the pattern, because we want to know - * about the block even if we end up skipping the prefetch. Otherwise we'd - * not be able to detect longer sequential pattens - we'd skip one block - * but then fail to skip the next couple blocks even in a perfect - * sequential pattern. This ocillation might even prevent the OS - * read-ahead from kicking in. - */ - prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; - prefetch->blockIndex++; - - /* - * Check if the last couple blocks are in a sequential pattern. We look - * for a sequential pattern of PREFETCH_SEQ_PATTERN_BLOCKS (4 by default), - * so we look for patterns of 5 pages (40kB) including the new block. - * - * XXX Perhaps this should be tied to effective_io_concurrency somehow? - * - * XXX Could it be harmful that we read the queue backwards? Maybe memory - * prefetching works better for the forward direction? - */ - for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) - { - /* - * Are there enough requests to confirm a sequential pattern? We only - * consider something to be sequential after finding a sequence of - * PREFETCH_SEQ_PATTERN_BLOCKS blocks. - * - * FIXME Better to move this outside the loop. - */ - if (prefetch->blockIndex < i) - return false; - - /* - * Calculate index of the earlier block (we need to do -1 as we - * already incremented the index when adding the new block to the - * queue). - */ - idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); - - /* - * For a sequential pattern, blocks "k" step ago needs to have block - * number by "k" smaller compared to the current block. - */ - if (prefetch->blockItems[idx] != (block - i)) - return false; - } - - return true; -} - -/* - * index_prefetch_add_cache - * Add a block to the cache, check if it was recently prefetched. - * - * We don't want to prefetch blocks that we already prefetched recently. It's - * cheap but not free, and the overhead may have measurable impact. - * - * This check needs to be very cheap, even with fairly large caches (hundreds - * of entries, see PREFETCH_CACHE_SIZE). - * - * A simple queue would allow expiring the requests, but checking if it - * contains a particular block prefetched would be expensive (linear search). - * Another option would be a simple hash table, which has fast lookup but - * does not allow expiring entries cheaply. - * - * The cache does not need to be perfect, we can accept false - * positives/negatives, as long as the rate is reasonably low. We also need - * to expire entries, so that only "recent" requests are remembered. - * - * We use a hybrid cache that is organized as many small LRU caches. Each - * block is mapped to a particular LRU by hashing (so it's a bit like a - * hash table). The LRU caches are tiny (e.g. 8 entries), and the expiration - * happens at the level of a single LRU (by tracking only the 8 most recent requests). - * - * This allows quick searches and expiration, but with false negatives (when a - * particular LRU has too many collisions, we may evict entries that are more - * recent than some other LRU). - * - * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 - * prefetch request in total (these are the default parameters.) - * - * The recency is determined using a prefetch counter, incremented every - * time we end up prefetching a block. The counter is uint64, so it should - * not wrap (125 zebibytes, would take ~4 million years at 1GB/s). - * - * To check if a block was prefetched recently, we calculate hash(block), - * and then linearly search if the tiny LRU has entry for the same block - * and request less than PREFETCH_CACHE_SIZE ago. - * - * At the same time, we either update the entry (for the queried block) if - * found, or replace the oldest/empty entry. - * - * If the block was not recently prefetched (i.e. we want to prefetch it), - * we increment the counter. - * - * Returns true if the block was recently prefetched (and thus we don't - * need to prefetch it again), or false (should do a prefetch). - * - * XXX It's a bit confusing these return values are inverse compared to - * what index_prefetch_is_sequential does. - */ -static bool -index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) -{ - PrefetchCacheEntry *entry; - - /* map the block number the the LRU */ - int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; - - /* age/index of the oldest entry in the LRU, to maybe use */ - uint64 oldestRequest = PG_UINT64_MAX; - int oldestIndex = -1; - - /* - * First add the block to the (tiny) top-level LRU cache and see if it's - * part of a sequential pattern. In this case we just ignore the block and - * don't prefetch it - we expect read-ahead to do a better job. - * - * XXX Maybe we should still add the block to the hybrid cache, in case we - * happen to access it later? That might help if we first scan a lot of - * the table sequentially, and then randomly. Not sure that's very likely - * with index access, though. - */ - if (index_prefetch_is_sequential(prefetch, block)) - { - prefetch->countSkipSequential++; - return true; - } - - /* - * See if we recently prefetched this block - we simply scan the LRU - * linearly. While doing that, we also track the oldest entry, so that we - * know where to put the block if we don't find a matching entry. - */ - for (int i = 0; i < PREFETCH_LRU_SIZE; i++) - { - entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; - - /* Is this the oldest prefetch request in this LRU? */ - if (entry->request < oldestRequest) - { - oldestRequest = entry->request; - oldestIndex = i; - } - - /* - * If the entry is unused (identified by request being set to 0), - * we're done. Notice the field is uint64, so empty entry is - * guaranteed to be the oldest one. - */ - if (entry->request == 0) - continue; - - /* Is this entry for the same block as the current request? */ - if (entry->block == block) - { - bool prefetched; - - /* - * Is the old request sufficiently recent? If yes, we treat the - * block as already prefetched. - * - * XXX We do add the cache size to the request in order not to - * have issues with uint64 underflows. - */ - prefetched = ((entry->request + PREFETCH_CACHE_SIZE) >= prefetch->prefetchReqNumber); - - /* Update the request number. */ - entry->request = ++prefetch->prefetchReqNumber; - - prefetch->countSkipCached += (prefetched) ? 1 : 0; - - return prefetched; - } - } - - /* - * We didn't find the block in the LRU, so store it either in an empty - * entry, or in the "oldest" prefetch request in this LRU. - */ - Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); - - /* FIXME do a nice macro */ - entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; - - entry->block = block; - entry->request = ++prefetch->prefetchReqNumber; - - /* not in the prefetch cache */ - return false; -} - /* * index_prefetch * Prefetch the TID, unless it's sequential or recently prefetched. @@ -1452,6 +1122,7 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) { IndexPrefetch prefetch = scan->xs_prefetch; BlockNumber block; + PrefetchBufferResult result; /* * No heap relation means bitmap index scan, which does prefetching at the @@ -1501,6 +1172,10 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) return; } + prefetch->countAll++; + + result = PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); + /* * Do not prefetch the same block over and over again, * @@ -1508,19 +1183,15 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) * to a sequence ID). It's not expensive (the block is in page cache * already, so no I/O), but it's not free either. */ - if (!index_prefetch_add_cache(prefetch, block)) + if (result.initiated_io) { prefetch->countPrefetch++; - - PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); pgBufferUsage.blks_prefetches++; } - - prefetch->countAll++; } /* ---------------- - * index_getnext_tid_prefetch - get the next TID from a scan + * index_getnext_tid - get the next TID from a scan * * The result is the next TID satisfying the scan keys, * or NULL if no more matching tuples exist. @@ -1529,9 +1200,20 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) * ---------------- */ ItemPointer -index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) +index_getnext_tid(IndexScanDesc scan, ScanDirection direction) +{ + /* Do prefetching (if requested/enabled). */ + index_prefetch_tids(scan, direction); + + /* Read the TID from the queue (or directly from the index). */ + return index_prefetch_get_tid(scan, direction); +} + +static void +index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) { - IndexPrefetch prefetch = scan->xs_prefetch; /* for convenience */ + /* for convenience */ + IndexPrefetch prefetch = scan->xs_prefetch; /* * If the prefetching is still active (i.e. enabled and we still @@ -1560,7 +1242,7 @@ index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) ItemPointer tid; /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + tid = index_getnext_tid_internal(scan, direction); /* * If we're out of index entries, we're done (and we mark the @@ -1583,9 +1265,16 @@ index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) * XXX index_getnext_tid_prefetch is only called for IOS (for now), * so skip prefetching of all-visible pages. */ - index_prefetch(scan, tid, true); + index_prefetch(scan, tid, scan->indexonly); } } +} + +static ItemPointer +index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction) +{ + /* for convenience */ + IndexPrefetch prefetch = scan->xs_prefetch; /* * With prefetching enabled (even if we already finished reading @@ -1607,7 +1296,7 @@ index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) ItemPointer tid; /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + tid = index_getnext_tid_internal(scan, direction); /* If we're out of index entries, we're done */ if (tid == NULL) @@ -1616,6 +1305,5 @@ index_getnext_tid_prefetch(IndexScanDesc scan, ScanDirection direction) Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); } - /* Return the TID of the tuple we found. */ return &scan->xs_heaptid; } diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 855afd5ba76..545046e98ad 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -120,6 +120,12 @@ IndexOnlyNext(IndexOnlyScanState *node) node->ioss_NumOrderByKeys, prefetch_max); + /* + * Remember this is index-only scan, because of prefetching. Not the most + * elegant way to pass this info. + */ + scandesc->indexonly = true; + node->ioss_ScanDesc = scandesc; @@ -142,7 +148,7 @@ IndexOnlyNext(IndexOnlyScanState *node) /* * OK, now that we have what we need, fetch the next tuple. */ - while ((tid = index_getnext_tid_prefetch(scandesc, direction)) != NULL) + while ((tid = index_getnext_tid(scandesc, direction)) != NULL) { bool tuple_from_heap = false; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index e7b915d6ce7..9f33796fd29 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -235,38 +235,6 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); -/* - * Cache of recently prefetched blocks, organized as a hash table of - * small LRU caches. Doesn't need to be perfectly accurate, but we - * aim to make false positives/negatives reasonably low. - */ -typedef struct PrefetchCacheEntry { - BlockNumber block; - uint64 request; -} PrefetchCacheEntry; - -/* - * Size of the cache of recently prefetched blocks - shouldn't be too - * small or too large. 1024 seems about right, it covers ~8MB of data. - * It's somewhat arbitrary, there's no particular formula saying it - * should not be higher/lower. - * - * The cache is structured as an array of small LRU caches, so the total - * size needs to be a multiple of LRU size. The LRU should be tiny to - * keep linear search cheap enough. - * - * XXX Maybe we could consider effective_cache_size or something? - */ -#define PREFETCH_LRU_SIZE 8 -#define PREFETCH_LRU_COUNT 128 -#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) - -/* - * Used to detect sequential patterns (and disable prefetching). - */ -#define PREFETCH_QUEUE_HISTORY 8 -#define PREFETCH_SEQ_PATTERN_BLOCKS 4 - typedef struct IndexPrefetchData { @@ -296,27 +264,6 @@ typedef struct IndexPrefetchData uint64 queueIndex; /* next TID to prefetch */ uint64 queueStart; /* first valid TID in queue */ uint64 queueEnd; /* first invalid (empty) TID in queue */ - - /* - * A couple of last prefetched blocks, used to check for certain access - * pattern and skip prefetching - e.g. for sequential access). - * - * XXX Separate from the main queue, because we only want to compare the - * block numbers, not the whole TID. In sequential access it's likely we - * read many items from each page, and we don't want to check many items - * (as that is much more expensive). - */ - BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; - uint64 blockIndex; /* index in the block (points to the first - * empty entry)*/ - - /* - * Cache of recently prefetched blocks, organized as a hash table of - * small LRU caches. - */ - uint64 prefetchReqNumber; - PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; - } IndexPrefetchData; #define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index 231a30ecc46..d5903492c6e 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -135,6 +135,7 @@ typedef struct IndexScanDescData bool ignore_killed_tuples; /* do not return killed entries */ bool xactStartedInRecovery; /* prevents killing/seeing killed * tuples */ + bool indexonly; /* is this index-only scan? */ /* index access method's private state */ void *opaque; /* access-method-specific info */ -- 2.42.0 [text/x-patch] v20231124-0003-check-page-cache-using-preadv2.patch (7.9K, 4-v20231124-0003-check-page-cache-using-preadv2.patch) download | inline diff: From 3a25a7534b4fc01bfc0043f027db922ab0b531fb Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Wed, 22 Nov 2023 18:21:45 +0100 Subject: [PATCH v20231124 3/7] check page cache using preadv2 Call preadv2 with NOWAIT flag, to check if a block already exists in page cache. --- src/backend/storage/buffer/bufmgr.c | 12 +++++++++ src/backend/storage/file/fd.c | 40 +++++++++++++++++++++++++++++ src/backend/storage/smgr/md.c | 27 +++++++++++++++++++ src/backend/storage/smgr/smgr.c | 19 ++++++++++++++ src/include/storage/fd.h | 1 + src/include/storage/md.h | 2 ++ src/include/storage/smgr.h | 2 ++ 7 files changed, 103 insertions(+) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index f7c67d504cd..74da9c1376b 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -567,8 +567,20 @@ PrefetchSharedBuffer(SMgrRelation smgr_reln, /* * Try to initiate an asynchronous read. This returns false in * recovery if the relation file doesn't exist. + * + * But first check if the block is already present in page cache. + * + * FIXME This breaks prefetch from recovery. Apparently that expects + * the prefetch to initiate the I/O, otherwise it fails with. But + * XLogPrefetcherNextBlock checks initiated_io, and may fail with: + * + * FATAL: could not prefetch relation 1663/16384/16401 block 83758 + * + * So maybe just fake the initiated_io=true in this case? Or not do + * this when in recovery. */ if ((io_direct_flags & IO_DIRECT_DATA) == 0 && + !smgrcached(smgr_reln, forkNum, blockNum) && smgrprefetch(smgr_reln, forkNum, blockNum)) { result.initiated_io = true; diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index f691ba09321..2c51a3376f3 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -78,6 +78,7 @@ #include <sys/resource.h> /* for getrlimit */ #include <sys/stat.h> #include <sys/types.h> +#include <sys/uio.h> #ifndef WIN32 #include <sys/mman.h> #endif @@ -2083,6 +2084,45 @@ retry: #endif } +/* + * FileCached - check if a given range of the file is in page cache. + * + * XXX relies on preadv2, probably needs to be checked by configure + */ +bool +FileCached(File file, off_t offset, off_t amount, uint32 wait_event_info) +{ +#if defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_WILLNEED) + int returnCode; + size_t readlen; + char buffer[BLCKSZ]; + struct iovec iov[1]; + + Assert(FileIsValid(file)); + + DO_DB(elog(LOG, "FilePrefetch: %d (%s) " INT64_FORMAT " " INT64_FORMAT, + file, VfdCache[file].fileName, + (int64) offset, (int64) amount)); + + returnCode = FileAccess(file); + if (returnCode < 0) + return false; + + /* XXX not sure if this ensures proper buffer alignment */ + iov[0].iov_base = &buffer; + iov[0].iov_len = amount; + + pgstat_report_wait_start(wait_event_info); + readlen = preadv2(VfdCache[file].fd, iov, 1, offset, RWF_NOWAIT); + pgstat_report_wait_end(); + + return (readlen == amount); +#else + Assert(FileIsValid(file)); + return false; +#endif +} + void FileWriteback(File file, off_t offset, off_t nbytes, uint32 wait_event_info) { diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c index fdecbad1709..16a7c424683 100644 --- a/src/backend/storage/smgr/md.c +++ b/src/backend/storage/smgr/md.c @@ -736,6 +736,33 @@ mdprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) return true; } +/* + * mdcached() -- Check if the whole block is already available in page cache. + */ +bool +mdcached(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) +{ +#ifdef USE_PREFETCH + off_t seekpos; + MdfdVec *v; + + Assert((io_direct_flags & IO_DIRECT_DATA) == 0); + + v = _mdfd_getseg(reln, forknum, blocknum, false, + InRecovery ? EXTENSION_RETURN_NULL : EXTENSION_FAIL); + if (v == NULL) + return false; + + seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE)); + + Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE); + + (void) FileCached(v->mdfd_vfd, seekpos, BLCKSZ, WAIT_EVENT_DATA_FILE_PREFETCH); +#endif /* USE_PREFETCH */ + + return true; +} + /* * mdread() -- Read the specified block from a relation. */ diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c index 5d0f3d515c3..209518aae01 100644 --- a/src/backend/storage/smgr/smgr.c +++ b/src/backend/storage/smgr/smgr.c @@ -55,6 +55,8 @@ typedef struct f_smgr BlockNumber blocknum, int nblocks, bool skipFsync); bool (*smgr_prefetch) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum); + bool (*smgr_cached) (SMgrRelation reln, ForkNumber forknum, + BlockNumber blocknum); void (*smgr_read) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, void *buffer); void (*smgr_write) (SMgrRelation reln, ForkNumber forknum, @@ -80,6 +82,7 @@ static const f_smgr smgrsw[] = { .smgr_extend = mdextend, .smgr_zeroextend = mdzeroextend, .smgr_prefetch = mdprefetch, + .smgr_cached = mdcached, .smgr_read = mdread, .smgr_write = mdwrite, .smgr_writeback = mdwriteback, @@ -550,6 +553,22 @@ smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum); } +/* + * smgrcached() -- Check if the specified block is already in page cache. + */ +bool +smgrcached(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) +{ + /* + * In recovery we consider the blocks not cached, so that PrefetchSharedBuffer + * initiates the I/O. XLogPrefetcherNextBlock relies on that. + */ + if (InRecovery) + return false; + + return smgrsw[reln->smgr_which].smgr_cached(reln, forknum, blocknum); +} + /* * smgrread() -- read a particular block from a relation into the supplied * buffer. diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index d9d5d9da5fb..c96a24dddd3 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -105,6 +105,7 @@ extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fil extern File OpenTemporaryFile(bool interXact); extern void FileClose(File file); extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info); +extern bool FileCached(File file, off_t offset, off_t amount, uint32 wait_event_info); extern int FileRead(File file, void *buffer, size_t amount, off_t offset, uint32 wait_event_info); extern int FileWrite(File file, const void *buffer, size_t amount, off_t offset, uint32 wait_event_info); extern int FileSync(File file, uint32 wait_event_info); diff --git a/src/include/storage/md.h b/src/include/storage/md.h index 941879ee6a8..8dc1382471e 100644 --- a/src/include/storage/md.h +++ b/src/include/storage/md.h @@ -32,6 +32,8 @@ extern void mdzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, int nblocks, bool skipFsync); extern bool mdprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum); +extern bool mdcached(SMgrRelation reln, ForkNumber forknum, + BlockNumber blocknum); extern void mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, void *buffer); extern void mdwrite(SMgrRelation reln, ForkNumber forknum, diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h index a9a179aabac..7fbed2a4291 100644 --- a/src/include/storage/smgr.h +++ b/src/include/storage/smgr.h @@ -96,6 +96,8 @@ extern void smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, int nblocks, bool skipFsync); extern bool smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum); +extern bool smgrcached(SMgrRelation reln, ForkNumber forknum, + BlockNumber blocknum); extern void smgrread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, void *buffer); extern void smgrwrite(SMgrRelation reln, ForkNumber forknum, -- 2.42.0 [text/x-patch] v20231124-0004-reintroduce-the-LRU-cache-of-recent-blocks.patch (8.2K, 5-v20231124-0004-reintroduce-the-LRU-cache-of-recent-blocks.patch) download | inline diff: From 8571e2b0ee705ea7d1b8d2f285c361ea010e4d2d Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Sat, 18 Nov 2023 12:05:53 +0100 Subject: [PATCH v20231124 4/7] reintroduce the LRU cache of recent blocks Useful for detecting sequential patterns, for which read-ahead works better than our prefetching, and checking shared buffers and page cache is not sufficient. --- src/backend/access/index/indexam.c | 137 +++++++++++++++++++++++++++++ src/include/access/genam.h | 30 +++++++ 2 files changed, 167 insertions(+) diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 54a704338f1..7456a69ab34 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -1082,6 +1082,125 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, return build_local_reloptions(&relopts, attoptions, validate); } +/* + * index_prefetch_is_sequential + * Track the block number and check if the I/O pattern is sequential, + * or if the same block was just prefetched. + * + * Prefetching is cheap, but for some access patterns the benefits are small + * compared to the extra overhead. In particular, for sequential access the + * read-ahead performed by the OS is very effective/efficient. Doing more + * prefetching is just increasing the costs. + * + * This tries to identify simple sequential patterns, so that we can skip + * the prefetching request. This is implemented by having a small queue + * of block numbers, and checking it before prefetching another block. + * + * We look at the preceding PREFETCH_SEQ_PATTERN_BLOCKS blocks, and see if + * they are sequential. We also check if the block is the same as the last + * request (which is not sequential). + * + * Note that the main prefetch queue is not really useful for this, as it + * stores TIDs while we care about block numbers. Consider a sorted table, + * with a perfectly sequential pattern when accessed through an index. Each + * heap page may have dozens of TIDs, but we need to check block numbers. + * We could keep enough TIDs to cover enough blocks, but then we also need + * to walk those when checking the pattern (in hot path). + * + * So instead, we maintain a small separate queue of block numbers, and we use + * this instead. + * + * Returns true if the block is in a sequential pattern (and so should not be + * prefetched), or false (not sequential, should be prefetched). + * + * XXX The name is a bit misleading, as it also adds the block number to the + * block queue and checks if the block is the same as the last one (which + * does not require a sequential pattern). + */ +static bool +index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) +{ + int idx; + + /* + * If the block queue is empty, just store the block and we're done (it's + * neither a sequential pattern, neither recently prefetched block). + */ + if (prefetch->blockIndex == 0) + { + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + return false; + } + + /* + * Check if it's the same as the immediately preceding block. We don't + * want to prefetch the same block over and over (which would happen for + * well correlated indexes). + * + * In principle we could rely on index_prefetch_add_cache doing this using + * the full cache, but this check is much cheaper and we need to look at + * the preceding block anyway, so we just do it. + * + * XXX Notice we haven't added the block to the block queue yet, and there + * is a preceding block (i.e. blockIndex-1 is valid). + */ + if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) + return true; + + /* + * Add the block number to the queue. + * + * We do this before checking if the pattern, because we want to know + * about the block even if we end up skipping the prefetch. Otherwise we'd + * not be able to detect longer sequential pattens - we'd skip one block + * but then fail to skip the next couple blocks even in a perfect + * sequential pattern. This ocillation might even prevent the OS + * read-ahead from kicking in. + */ + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + + /* + * Check if the last couple blocks are in a sequential pattern. We look + * for a sequential pattern of PREFETCH_SEQ_PATTERN_BLOCKS (4 by default), + * so we look for patterns of 5 pages (40kB) including the new block. + * + * XXX Perhaps this should be tied to effective_io_concurrency somehow? + * + * XXX Could it be harmful that we read the queue backwards? Maybe memory + * prefetching works better for the forward direction? + */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) + { + /* + * Are there enough requests to confirm a sequential pattern? We only + * consider something to be sequential after finding a sequence of + * PREFETCH_SEQ_PATTERN_BLOCKS blocks. + * + * FIXME Better to move this outside the loop. + */ + if (prefetch->blockIndex < i) + return false; + + /* + * Calculate index of the earlier block (we need to do -1 as we + * already incremented the index when adding the new block to the + * queue). + */ + idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); + + /* + * For a sequential pattern, blocks "k" step ago needs to have block + * number by "k" smaller compared to the current block. + */ + if (prefetch->blockItems[idx] != (block - i)) + return false; + } + + return true; +} + /* * index_prefetch * Prefetch the TID, unless it's sequential or recently prefetched. @@ -1172,8 +1291,26 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) return; } + /* + * Now also check if the blocks to prefetch are in a sequential pattern. + * We do it here because we need to do this check before PrefetchBuffer + * initiates the prefetch, and we it can't do this easily (as it doesn't + * know in what context it's called in). So we do it here. + * + * We use a tiny LRU cache and see if the blocks follow a sequential + * pattern - if it's the same as the previous block, or if the last + * couple blocks are a continguous sequence, we don't prefetch it. + */ + if (index_prefetch_is_sequential(prefetch, block)) + { + prefetch->countSkipSequential++; + return; + } + + /* XXX shouldn't this be before the VM / sequenqial check? */ prefetch->countAll++; + /* OK, try prefetching the block. */ result = PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); /* diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 9f33796fd29..9e2d77ef23b 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -235,6 +235,22 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); +/* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. Doesn't need to be perfectly accurate, but we + * aim to make false positives/negatives reasonably low. + */ +typedef struct PrefetchCacheEntry { + BlockNumber block; + uint64 request; +} PrefetchCacheEntry; + +/* + * Used to detect sequential patterns (to not prefetch in this case). + */ +#define PREFETCH_QUEUE_HISTORY 8 +#define PREFETCH_SEQ_PATTERN_BLOCKS 4 + typedef struct IndexPrefetchData { @@ -264,6 +280,20 @@ typedef struct IndexPrefetchData uint64 queueIndex; /* next TID to prefetch */ uint64 queueStart; /* first valid TID in queue */ uint64 queueEnd; /* first invalid (empty) TID in queue */ + + /* + * A couple of last prefetched blocks, used to check for certain access + * pattern and skip prefetching - e.g. for sequential access). + * + * XXX Separate from the main queue, because we only want to compare the + * block numbers, not the whole TID. In sequential access it's likely we + * read many items from each page, and we don't want to check many items + * (as that is much more expensive). + */ + BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; + uint64 blockIndex; /* index in the block (points to the first + * empty entry)*/ + } IndexPrefetchData; #define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) -- 2.42.0 [text/x-patch] v20231124-0005-hold-the-vm-buffer-for-IOS-prefetching.patch (2.4K, 6-v20231124-0005-hold-the-vm-buffer-for-IOS-prefetching.patch) download | inline diff: From 23eead34921a16b8301160ce8bde51be615856f0 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Wed, 22 Nov 2023 23:26:08 +0100 Subject: [PATCH v20231124 5/7] hold the vm buffer for IOS prefetching --- src/backend/access/index/indexam.c | 7 ++----- src/backend/executor/nodeIndexonlyscan.c | 12 ++++++++++++ src/include/access/genam.h | 3 +++ 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 7456a69ab34..53948986ada 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -336,6 +336,7 @@ index_beginscan_internal(Relation indexRelation, prefetcher->prefetchTarget = 0; prefetcher->prefetchMaxTarget = prefetch_max; + prefetcher->vmBuffer = InvalidBuffer; scan->xs_prefetch = prefetcher; } @@ -1278,14 +1279,10 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) if (skip_all_visible) { bool all_visible; - Buffer vmbuffer = InvalidBuffer; all_visible = VM_ALL_VISIBLE(scan->heapRelation, block, - &vmbuffer); - - if (vmbuffer != InvalidBuffer) - ReleaseBuffer(vmbuffer); + &prefetch->vmBuffer); if (all_visible) return; diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 545046e98ad..60cb772344f 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -412,6 +412,18 @@ ExecEndIndexOnlyScan(IndexOnlyScanState *node) node->ioss_VMBuffer = InvalidBuffer; } + /* Release VM buffer pin from prefetcher, if any. */ + if (indexScanDesc && indexScanDesc->xs_prefetch) + { + IndexPrefetch indexPrefetch = indexScanDesc->xs_prefetch; + + if (indexPrefetch->vmBuffer != InvalidBuffer) + { + ReleaseBuffer(indexPrefetch->vmBuffer); + indexPrefetch->vmBuffer = InvalidBuffer; + } + } + /* * close the index relation (no-op if we didn't open it) */ diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 9e2d77ef23b..8a3be673730 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -270,6 +270,9 @@ typedef struct IndexPrefetchData uint64 countSkipSequential; uint64 countSkipCached; + /* used when prefetching index-only scans */ + Buffer vmBuffer; + /* * Queue of TIDs to prefetch. * -- 2.42.0 [text/x-patch] v20231124-0006-poc-reuse-vm-information.patch (8.8K, 7-v20231124-0006-poc-reuse-vm-information.patch) download | inline diff: From 58a3cea898f810aa4646e6327c5e0241e6886e66 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Thu, 23 Nov 2023 00:47:17 +0100 Subject: [PATCH v20231124 6/7] poc: reuse vm information --- src/backend/access/index/indexam.c | 59 ++++++++++++++++-------- src/backend/executor/nodeIndexonlyscan.c | 8 +++- src/include/access/genam.h | 12 +++-- 3 files changed, 56 insertions(+), 23 deletions(-) diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 53948986ada..45c97aff5ef 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -113,8 +113,8 @@ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int prefetch_max); static void index_prefetch_tids(IndexScanDesc scan, ScanDirection direction); -static ItemPointer index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction); -static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible); +static ItemPointer index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_visible); +static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible); /* ---------------------------------------------------------------- @@ -721,10 +721,11 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot * if (!scan->xs_heap_continue) { - ItemPointer tid; + ItemPointer tid; + bool all_visible; /* Time to fetch the next TID from the index */ - tid = index_prefetch_get_tid(scan, direction); + tid = index_prefetch_get_tid(scan, direction, &all_visible); /* If we're out of index entries, we're done */ if (tid == NULL) @@ -1238,12 +1239,15 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) * in BTScanPosData.nextPage. */ static void -index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) +index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible) { IndexPrefetch prefetch = scan->xs_prefetch; BlockNumber block; PrefetchBufferResult result; + /* by default not all visible (or we didn't check) */ + *all_visible = false; + /* * No heap relation means bitmap index scan, which does prefetching at the * bitmap heap scan, so no prefetch here (we can't do it anyway, without @@ -1275,16 +1279,19 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) * we can propagate it here). Or at least do it for a bulk of prefetches, * although that's not very useful - after the ramp-up we will prefetch * the pages one by one anyway. + * + * XXX Ideally we'd also propagate this to the executor, so that the + * nodeIndexonlyscan.c doesn't need to repeat the same VM check (which + * is measurable). But the index_getnext_tid() is not really well + * suited for that, so the API needs a change.s */ if (skip_all_visible) { - bool all_visible; + *all_visible = VM_ALL_VISIBLE(scan->heapRelation, + block, + &prefetch->vmBuffer); - all_visible = VM_ALL_VISIBLE(scan->heapRelation, - block, - &prefetch->vmBuffer); - - if (all_visible) + if (*all_visible) return; } @@ -1336,11 +1343,23 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible) ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction) { + bool all_visible; /* ignored */ + /* Do prefetching (if requested/enabled). */ index_prefetch_tids(scan, direction); /* Read the TID from the queue (or directly from the index). */ - return index_prefetch_get_tid(scan, direction); + return index_prefetch_get_tid(scan, direction, &all_visible); +} + +ItemPointer +index_getnext_tid_vm(IndexScanDesc scan, ScanDirection direction, bool *all_visible) +{ + /* Do prefetching (if requested/enabled). */ + index_prefetch_tids(scan, direction); + + /* Read the TID from the queue (or directly from the index). */ + return index_prefetch_get_tid(scan, direction, all_visible); } static void @@ -1374,6 +1393,7 @@ index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) while (!PREFETCH_FULL(prefetch)) { ItemPointer tid; + bool all_visible; /* Time to fetch the next TID from the index */ tid = index_getnext_tid_internal(scan, direction); @@ -1390,22 +1410,23 @@ index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); - prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)] = *tid; - prefetch->queueEnd++; - /* * Issue the actuall prefetch requests for the new TID. * * XXX index_getnext_tid_prefetch is only called for IOS (for now), * so skip prefetching of all-visible pages. */ - index_prefetch(scan, tid, scan->indexonly); + index_prefetch(scan, tid, scan->indexonly, &all_visible); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)].tid = *tid; + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)].all_visible = all_visible; + prefetch->queueEnd++; } } } static ItemPointer -index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction) +index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_visible) { /* for convenience */ IndexPrefetch prefetch = scan->xs_prefetch; @@ -1422,7 +1443,8 @@ index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction) if (PREFETCH_DONE(prefetch)) return NULL; - scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)]; + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)].tid; + *all_visible = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)].all_visible; prefetch->queueIndex++; } else /* not prefetching, just do the regular work */ @@ -1431,6 +1453,7 @@ index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction) /* Time to fetch the next TID from the index */ tid = index_getnext_tid_internal(scan, direction); + *all_visible = false; /* If we're out of index entries, we're done */ if (tid == NULL) diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 60cb772344f..b6660c10a63 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -66,6 +66,7 @@ IndexOnlyNext(IndexOnlyScanState *node) TupleTableSlot *slot; ItemPointer tid; Relation heapRel = node->ss.ss_currentRelation; + bool all_visible; /* * extract necessary information from index scan node @@ -148,7 +149,7 @@ IndexOnlyNext(IndexOnlyScanState *node) /* * OK, now that we have what we need, fetch the next tuple. */ - while ((tid = index_getnext_tid(scandesc, direction)) != NULL) + while ((tid = index_getnext_tid_vm(scandesc, direction, &all_visible)) != NULL) { bool tuple_from_heap = false; @@ -187,8 +188,11 @@ IndexOnlyNext(IndexOnlyScanState *node) * * It's worth going through this complexity to avoid needing to lock * the VM buffer, which could cause significant contention. + * + * XXX Skip if we already know the page is all visible from prefetcher. */ - if (!VM_ALL_VISIBLE(scandesc->heapRelation, + if (!all_visible && + !VM_ALL_VISIBLE(scandesc->heapRelation, ItemPointerGetBlockNumber(tid), &node->ioss_VMBuffer)) { diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 8a3be673730..db1dc9c44b6 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -175,8 +175,9 @@ extern IndexScanDesc index_beginscan_parallel(Relation heaprel, int prefetch_max); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); -extern ItemPointer index_getnext_tid_prefetch(IndexScanDesc scan, - ScanDirection direction); +extern ItemPointer index_getnext_tid_vm(IndexScanDesc scan, + ScanDirection direction, + bool *all_visible); struct TupleTableSlot; extern bool index_fetch_heap(IndexScanDesc scan, struct TupleTableSlot *slot); extern bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, @@ -251,6 +252,11 @@ typedef struct PrefetchCacheEntry { #define PREFETCH_QUEUE_HISTORY 8 #define PREFETCH_SEQ_PATTERN_BLOCKS 4 +typedef struct PrefetchEntry +{ + ItemPointerData tid; + bool all_visible; +} PrefetchEntry; typedef struct IndexPrefetchData { @@ -279,7 +285,7 @@ typedef struct IndexPrefetchData * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler * than dynamically adjusting for custom values. */ - ItemPointerData queueItems[MAX_IO_CONCURRENCY]; + PrefetchEntry queueItems[MAX_IO_CONCURRENCY]; uint64 queueIndex; /* next TID to prefetch */ uint64 queueStart; /* first valid TID in queue */ uint64 queueEnd; /* first invalid (empty) TID in queue */ -- 2.42.0 [text/x-patch] v20231124-0007-20231016-reworked.patch (18.7K, 8-v20231124-0007-20231016-reworked.patch) download | inline diff: From 3ba126559c69a31f6f0c48c85de2c892cebac4f3 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Fri, 24 Nov 2023 12:31:43 +0100 Subject: [PATCH v20231124 7/7] 20231016-reworked --- src/backend/access/index/indexam.c | 195 ++++++++++++++++++++++++---- src/backend/storage/buffer/bufmgr.c | 12 -- src/backend/storage/file/fd.c | 40 ------ src/backend/storage/smgr/md.c | 27 ---- src/backend/storage/smgr/smgr.c | 19 --- src/include/access/genam.h | 25 +++- src/include/storage/fd.h | 1 - src/include/storage/md.h | 2 - src/include/storage/smgr.h | 2 - 9 files changed, 195 insertions(+), 128 deletions(-) diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index 45c97aff5ef..82e3266bcc0 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -1203,6 +1203,148 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) return true; } +/* + * index_prefetch_add_cache + * Add a block to the cache, check if it was recently prefetched. + * + * We don't want to prefetch blocks that we already prefetched recently. It's + * cheap but not free, and the overhead may have measurable impact. + * + * This check needs to be very cheap, even with fairly large caches (hundreds + * of entries, see PREFETCH_CACHE_SIZE). + * + * A simple queue would allow expiring the requests, but checking if it + * contains a particular block prefetched would be expensive (linear search). + * Another option would be a simple hash table, which has fast lookup but + * does not allow expiring entries cheaply. + * + * The cache does not need to be perfect, we can accept false + * positives/negatives, as long as the rate is reasonably low. We also need + * to expire entries, so that only "recent" requests are remembered. + * + * We use a hybrid cache that is organized as many small LRU caches. Each + * block is mapped to a particular LRU by hashing (so it's a bit like a + * hash table). The LRU caches are tiny (e.g. 8 entries), and the expiration + * happens at the level of a single LRU (by tracking only the 8 most recent requests). + * + * This allows quick searches and expiration, but with false negatives (when a + * particular LRU has too many collisions, we may evict entries that are more + * recent than some other LRU). + * + * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 + * prefetch request in total (these are the default parameters.) + * + * The recency is determined using a prefetch counter, incremented every + * time we end up prefetching a block. The counter is uint64, so it should + * not wrap (125 zebibytes, would take ~4 million years at 1GB/s). + * + * To check if a block was prefetched recently, we calculate hash(block), + * and then linearly search if the tiny LRU has entry for the same block + * and request less than PREFETCH_CACHE_SIZE ago. + * + * At the same time, we either update the entry (for the queried block) if + * found, or replace the oldest/empty entry. + * + * If the block was not recently prefetched (i.e. we want to prefetch it), + * we increment the counter. + * + * Returns true if the block was recently prefetched (and thus we don't + * need to prefetch it again), or false (should do a prefetch). + * + * XXX It's a bit confusing these return values are inverse compared to + * what index_prefetch_is_sequential does. + */ +static bool +index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) +{ + PrefetchCacheEntry *entry; + + /* map the block number the the LRU */ + int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; + + /* age/index of the oldest entry in the LRU, to maybe use */ + uint64 oldestRequest = PG_UINT64_MAX; + int oldestIndex = -1; + + /* + * First add the block to the (tiny) top-level LRU cache and see if it's + * part of a sequential pattern. In this case we just ignore the block and + * don't prefetch it - we expect read-ahead to do a better job. + * + * XXX Maybe we should still add the block to the hybrid cache, in case we + * happen to access it later? That might help if we first scan a lot of + * the table sequentially, and then randomly. Not sure that's very likely + * with index access, though. + */ + if (index_prefetch_is_sequential(prefetch, block)) + { + prefetch->countSkipSequential++; + return true; + } + + /* + * See if we recently prefetched this block - we simply scan the LRU + * linearly. While doing that, we also track the oldest entry, so that we + * know where to put the block if we don't find a matching entry. + */ + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) + { + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; + + /* Is this the oldest prefetch request in this LRU? */ + if (entry->request < oldestRequest) + { + oldestRequest = entry->request; + oldestIndex = i; + } + + /* + * If the entry is unused (identified by request being set to 0), + * we're done. Notice the field is uint64, so empty entry is + * guaranteed to be the oldest one. + */ + if (entry->request == 0) + continue; + + /* Is this entry for the same block as the current request? */ + if (entry->block == block) + { + bool prefetched; + + /* + * Is the old request sufficiently recent? If yes, we treat the + * block as already prefetched. + * + * XXX We do add the cache size to the request in order not to + * have issues with uint64 underflows. + */ + prefetched = ((entry->request + PREFETCH_CACHE_SIZE) >= prefetch->prefetchReqNumber); + + /* Update the request number. */ + entry->request = ++prefetch->prefetchReqNumber; + + prefetch->countSkipCached += (prefetched) ? 1 : 0; + + return prefetched; + } + } + + /* + * We didn't find the block in the LRU, so store it either in an empty + * entry, or in the "oldest" prefetch request in this LRU. + */ + Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); + + /* FIXME do a nice macro */ + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; + + entry->block = block; + entry->request = ++prefetch->prefetchReqNumber; + + /* not in the prefetch cache */ + return false; +} + /* * index_prefetch * Prefetch the TID, unless it's sequential or recently prefetched. @@ -1237,13 +1379,35 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) * * XXX Maybe we could/should also prefetch the next index block, e.g. stored * in BTScanPosData.nextPage. + * + * XXX Could we tune the cache size based on execution statistics? We have + * a cache of limited size (PREFETCH_CACHE_SIZE = 1024 by default), but + * how do we know it's the right size? Ideally, we'd have a cache large + * enough to track actually cached blocks. If the OS caches 10240 pages, + * then we may do 90% of prefetch requests unnecessarily. Or maybe there's + * a lot of contention, blocks are evicted quickly, and 90% of the blocks + * in the cache are not actually cached anymore? But we do have a concept + * of sequential request ID (PrefetchCacheEntry->request), which gives us + * information about "age" of the last prefetch. Now it's used only when + * evicting entries (to keep the more recent one), but maybe we could also + * use it when deciding if the page is cached. Right now any block that's + * in the cache is considered cached and not prefetched, but maybe we could + * have "max age", and tune it based on feedback from reading the blocks + * later. For example, if we find the block in cache and decide not to + * prefetch it, but then later find we have to do I/O, it means our cache + * is too large. And we could "reduce" the maximum age (measured from the + * current prefetchReqNumber value), so that only more recent blocks would + * be considered cached. Not sure about the opposite direction, where we + * decide to prefetch a block - AFAIK we don't have a way to determine if + * I/O was needed or not in this case (so we can't increase the max age). + * But maybe we could di that somehow speculatively, i.e. increase the + * value once in a while, and see what happens. */ static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible) { IndexPrefetch prefetch = scan->xs_prefetch; BlockNumber block; - PrefetchBufferResult result; /* by default not all visible (or we didn't check) */ *all_visible = false; @@ -1295,28 +1459,6 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool return; } - /* - * Now also check if the blocks to prefetch are in a sequential pattern. - * We do it here because we need to do this check before PrefetchBuffer - * initiates the prefetch, and we it can't do this easily (as it doesn't - * know in what context it's called in). So we do it here. - * - * We use a tiny LRU cache and see if the blocks follow a sequential - * pattern - if it's the same as the previous block, or if the last - * couple blocks are a continguous sequence, we don't prefetch it. - */ - if (index_prefetch_is_sequential(prefetch, block)) - { - prefetch->countSkipSequential++; - return; - } - - /* XXX shouldn't this be before the VM / sequenqial check? */ - prefetch->countAll++; - - /* OK, try prefetching the block. */ - result = PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); - /* * Do not prefetch the same block over and over again, * @@ -1324,11 +1466,15 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool * to a sequence ID). It's not expensive (the block is in page cache * already, so no I/O), but it's not free either. */ - if (result.initiated_io) + if (!index_prefetch_add_cache(prefetch, block)) { prefetch->countPrefetch++; + + PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); pgBufferUsage.blks_prefetches++; } + + prefetch->countAll++; } /* ---------------- @@ -1462,5 +1608,6 @@ index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_vi Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); } + /* Return the TID of the tuple we found. */ return &scan->xs_heaptid; } diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 74da9c1376b..f7c67d504cd 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -567,20 +567,8 @@ PrefetchSharedBuffer(SMgrRelation smgr_reln, /* * Try to initiate an asynchronous read. This returns false in * recovery if the relation file doesn't exist. - * - * But first check if the block is already present in page cache. - * - * FIXME This breaks prefetch from recovery. Apparently that expects - * the prefetch to initiate the I/O, otherwise it fails with. But - * XLogPrefetcherNextBlock checks initiated_io, and may fail with: - * - * FATAL: could not prefetch relation 1663/16384/16401 block 83758 - * - * So maybe just fake the initiated_io=true in this case? Or not do - * this when in recovery. */ if ((io_direct_flags & IO_DIRECT_DATA) == 0 && - !smgrcached(smgr_reln, forkNum, blockNum) && smgrprefetch(smgr_reln, forkNum, blockNum)) { result.initiated_io = true; diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 2c51a3376f3..f691ba09321 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -78,7 +78,6 @@ #include <sys/resource.h> /* for getrlimit */ #include <sys/stat.h> #include <sys/types.h> -#include <sys/uio.h> #ifndef WIN32 #include <sys/mman.h> #endif @@ -2084,45 +2083,6 @@ retry: #endif } -/* - * FileCached - check if a given range of the file is in page cache. - * - * XXX relies on preadv2, probably needs to be checked by configure - */ -bool -FileCached(File file, off_t offset, off_t amount, uint32 wait_event_info) -{ -#if defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_WILLNEED) - int returnCode; - size_t readlen; - char buffer[BLCKSZ]; - struct iovec iov[1]; - - Assert(FileIsValid(file)); - - DO_DB(elog(LOG, "FilePrefetch: %d (%s) " INT64_FORMAT " " INT64_FORMAT, - file, VfdCache[file].fileName, - (int64) offset, (int64) amount)); - - returnCode = FileAccess(file); - if (returnCode < 0) - return false; - - /* XXX not sure if this ensures proper buffer alignment */ - iov[0].iov_base = &buffer; - iov[0].iov_len = amount; - - pgstat_report_wait_start(wait_event_info); - readlen = preadv2(VfdCache[file].fd, iov, 1, offset, RWF_NOWAIT); - pgstat_report_wait_end(); - - return (readlen == amount); -#else - Assert(FileIsValid(file)); - return false; -#endif -} - void FileWriteback(File file, off_t offset, off_t nbytes, uint32 wait_event_info) { diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c index 16a7c424683..fdecbad1709 100644 --- a/src/backend/storage/smgr/md.c +++ b/src/backend/storage/smgr/md.c @@ -736,33 +736,6 @@ mdprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) return true; } -/* - * mdcached() -- Check if the whole block is already available in page cache. - */ -bool -mdcached(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) -{ -#ifdef USE_PREFETCH - off_t seekpos; - MdfdVec *v; - - Assert((io_direct_flags & IO_DIRECT_DATA) == 0); - - v = _mdfd_getseg(reln, forknum, blocknum, false, - InRecovery ? EXTENSION_RETURN_NULL : EXTENSION_FAIL); - if (v == NULL) - return false; - - seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE)); - - Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE); - - (void) FileCached(v->mdfd_vfd, seekpos, BLCKSZ, WAIT_EVENT_DATA_FILE_PREFETCH); -#endif /* USE_PREFETCH */ - - return true; -} - /* * mdread() -- Read the specified block from a relation. */ diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c index 209518aae01..5d0f3d515c3 100644 --- a/src/backend/storage/smgr/smgr.c +++ b/src/backend/storage/smgr/smgr.c @@ -55,8 +55,6 @@ typedef struct f_smgr BlockNumber blocknum, int nblocks, bool skipFsync); bool (*smgr_prefetch) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum); - bool (*smgr_cached) (SMgrRelation reln, ForkNumber forknum, - BlockNumber blocknum); void (*smgr_read) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, void *buffer); void (*smgr_write) (SMgrRelation reln, ForkNumber forknum, @@ -82,7 +80,6 @@ static const f_smgr smgrsw[] = { .smgr_extend = mdextend, .smgr_zeroextend = mdzeroextend, .smgr_prefetch = mdprefetch, - .smgr_cached = mdcached, .smgr_read = mdread, .smgr_write = mdwrite, .smgr_writeback = mdwriteback, @@ -553,22 +550,6 @@ smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum); } -/* - * smgrcached() -- Check if the specified block is already in page cache. - */ -bool -smgrcached(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum) -{ - /* - * In recovery we consider the blocks not cached, so that PrefetchSharedBuffer - * initiates the I/O. XLogPrefetcherNextBlock relies on that. - */ - if (InRecovery) - return false; - - return smgrsw[reln->smgr_which].smgr_cached(reln, forknum, blocknum); -} - /* * smgrread() -- read a particular block from a relation into the supplied * buffer. diff --git a/src/include/access/genam.h b/src/include/access/genam.h index db1dc9c44b6..b5dbe971770 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -247,7 +247,23 @@ typedef struct PrefetchCacheEntry { } PrefetchCacheEntry; /* - * Used to detect sequential patterns (to not prefetch in this case). + * Size of the cache of recently prefetched blocks - shouldn't be too + * small or too large. 1024 seems about right, it covers ~8MB of data. + * It's somewhat arbitrary, there's no particular formula saying it + * should not be higher/lower. + * + * The cache is structured as an array of small LRU caches, so the total + * size needs to be a multiple of LRU size. The LRU should be tiny to + * keep linear search cheap enough. + * + * XXX Maybe we could consider effective_cache_size or something? + */ +#define PREFETCH_LRU_SIZE 8 +#define PREFETCH_LRU_COUNT 128 +#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) + +/* + * Used to detect sequential patterns (and disable prefetching). */ #define PREFETCH_QUEUE_HISTORY 8 #define PREFETCH_SEQ_PATTERN_BLOCKS 4 @@ -303,6 +319,13 @@ typedef struct IndexPrefetchData uint64 blockIndex; /* index in the block (points to the first * empty entry)*/ + /* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. + */ + uint64 prefetchReqNumber; + PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; + } IndexPrefetchData; #define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h index c96a24dddd3..d9d5d9da5fb 100644 --- a/src/include/storage/fd.h +++ b/src/include/storage/fd.h @@ -105,7 +105,6 @@ extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fil extern File OpenTemporaryFile(bool interXact); extern void FileClose(File file); extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info); -extern bool FileCached(File file, off_t offset, off_t amount, uint32 wait_event_info); extern int FileRead(File file, void *buffer, size_t amount, off_t offset, uint32 wait_event_info); extern int FileWrite(File file, const void *buffer, size_t amount, off_t offset, uint32 wait_event_info); extern int FileSync(File file, uint32 wait_event_info); diff --git a/src/include/storage/md.h b/src/include/storage/md.h index 8dc1382471e..941879ee6a8 100644 --- a/src/include/storage/md.h +++ b/src/include/storage/md.h @@ -32,8 +32,6 @@ extern void mdzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, int nblocks, bool skipFsync); extern bool mdprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum); -extern bool mdcached(SMgrRelation reln, ForkNumber forknum, - BlockNumber blocknum); extern void mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, void *buffer); extern void mdwrite(SMgrRelation reln, ForkNumber forknum, diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h index 7fbed2a4291..a9a179aabac 100644 --- a/src/include/storage/smgr.h +++ b/src/include/storage/smgr.h @@ -96,8 +96,6 @@ extern void smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, int nblocks, bool skipFsync); extern bool smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum); -extern bool smgrcached(SMgrRelation reln, ForkNumber forknum, - BlockNumber blocknum); extern void smgrread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, void *buffer); extern void smgrwrite(SMgrRelation reln, ForkNumber forknum, -- 2.42.0 [image/png] point-0-ios-improvement-small.png (187.7K, 9-point-0-ios-improvement-small.png) download | view image [image/png] point-4-regressions-small.png (221.2K, 10-point-4-regressions-small.png) download | view image ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-12-09 18:08 Tomas Vondra <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Tomas Vondra @ 2023-12-09 18:08 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> Hi, Here's a simplified version of the patch series, with two important changes from the last version shared on 2023/11/24. Firstly, it abandons the idea to use preadv2() to check page cache. This initially seemed like a great way to check if prefetching is needed, but in practice it seems so expensive it's not really beneficial (especially in the "cached" case, which is where it matters most). Note: There's one more reason to not want rely on preadv2() that I forgot to mention - it's a Linux-specific thing. I wouldn't mind using it to improve already acceptable behavior, but it doesn't seem like a great idea if performance without would be poor. Secondly, this reworks multiple aspects of the "layering". Until now, the prefetching info was stored in IndexScanDesc and initialized in indexam.c in the various "beginscan" functions. That was obviously wrong - IndexScanDesc is just a description of what the scan should do, not a place where execution state (which the prefetch queue is) should be stored. IndexScanState (and IndexOnlyScanState) is a more appropriate place, so I moved it there. This also means the various "beginscan" functions don't need any changes (i.e. not even get prefetch_max), which is nice. Because the prefetch state is created/initialized elsewhere. But there's a layering problem that I don't know how to solve - I don't see how we could make indexam.c entirely oblivious to the prefetching, and move it entirely to the executor. Because how else would you know what to prefetch? With index_getnext_tid() I can imagine fetching XIDs ahead, stashing them into a queue, and prefetching based on that. That's kinda what the patch does, except that it does it from inside index_getnext_tid(). But that does not work for index_getnext_slot(), because that already reads the heap tuples. We could say prefetching only works for index_getnext_tid(), but that seems a bit weird because that's what regular index scans do. (There's a patch to evaluate filters on index, which switches index scans to index_getnext_tid(), so that'd make prefetching work too, but I'd ignore that here. There are other index_getnext_slot() callers, and I don't think we should accept does not work for those places seems wrong (e.g. execIndexing/execReplication would benefit from prefetching, I think). The patch just adds a "prefetcher" argument to index_getnext_*(), and the prefetching still happens there. I guess we could move most of the prefether typedefs/code somewhere, but I don't quite see how it could be done in executor entirely. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company Attachments: [text/x-patch] v20231209-0001-prefetch-2023-11-24.patch (56.3K, 2-v20231209-0001-prefetch-2023-11-24.patch) download | inline diff: From a3335da2a7a28dbb258380fa23d9ddd7c887f1d9 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Fri, 17 Nov 2023 23:54:19 +0100 Subject: [PATCH v20231209 1/2] prefetch 2023-11-24 Patch version shared on 2023/11/24. --- src/backend/access/heap/heapam_handler.c | 12 +- src/backend/access/index/genam.c | 31 +- src/backend/access/index/indexam.c | 645 ++++++++++++++++++++++- src/backend/commands/explain.c | 18 + src/backend/executor/execIndexing.c | 6 +- src/backend/executor/execReplication.c | 9 +- src/backend/executor/instrument.c | 4 + src/backend/executor/nodeIndexonlyscan.c | 97 +++- src/backend/executor/nodeIndexscan.c | 80 ++- src/backend/utils/adt/selfuncs.c | 3 +- src/include/access/genam.h | 110 +++- src/include/access/relscan.h | 10 + src/include/executor/instrument.h | 2 + 13 files changed, 997 insertions(+), 30 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 7c28dafb728..89474078951 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -44,6 +44,7 @@ #include "storage/smgr.h" #include "utils/builtins.h" #include "utils/rel.h" +#include "utils/spccache.h" static void reform_and_rewrite_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, @@ -747,6 +748,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, PROGRESS_CLUSTER_INDEX_RELID }; int64 ci_val[2]; + int prefetch_max; + + /* + * Get the prefetch target for the old tablespace (which is what we'll + * read using the index). We'll use it as a reset value too, although + * there should be no rescans for CLUSTER etc. + */ + prefetch_max = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; @@ -755,7 +764,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tableScan = NULL; heapScan = NULL; - indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0); + indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0, + prefetch_max); index_rescan(indexScan, NULL, 0, NULL, 0); } else diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 4ca12006843..d45a209ee3a 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -126,6 +126,9 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys) scan->xs_hitup = NULL; scan->xs_hitupdesc = NULL; + /* Information used for asynchronous prefetching during index scans. */ + scan->xs_prefetch = NULL; + return scan; } @@ -440,8 +443,20 @@ systable_beginscan(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* + * We don't do any prefetching on system catalogs, for two main reasons. + * + * Firstly, we usually do PK lookups, which makes prefetching pointles, + * or we often don't know how many rows to expect (and the numbers tend + * to be fairly low). So it's not clear it'd help. Furthermore, places + * that are sensitive tend to use syscache anyway. + * + * Secondly, we can't call get_tablespace_io_concurrency() because that + * does a sysscan internally, so it might lead to a cycle. We could use + * use effective_io_concurrency, but it doesn't seem worth it. + */ sysscan->iscan = index_beginscan(heapRelation, irel, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; } @@ -696,8 +711,20 @@ systable_beginscan_ordered(Relation heapRelation, elog(ERROR, "column is not in index"); } + /* + * We don't do any prefetching on system catalogs, for two main reasons. + * + * Firstly, we usually do PK lookups, which makes prefetching pointles, + * or we often don't know how many rows to expect (and the numbers tend + * to be fairly low). So it's not clear it'd help. Furthermore, places + * that are sensitive tend to use syscache anyway. + * + * Secondly, we can't call get_tablespace_io_concurrency() because that + * does a sysscan internally, so it might lead to a cycle. We could use + * use effective_io_concurrency, but it doesn't seem worth it. + */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, - snapshot, nkeys, 0); + snapshot, nkeys, 0, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index f23e0199f08..e493548b68a 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -49,16 +49,19 @@ #include "access/relscan.h" #include "access/tableam.h" #include "access/transam.h" +#include "access/visibilitymap.h" #include "access/xlog.h" #include "catalog/index.h" #include "catalog/pg_amproc.h" #include "catalog/pg_type.h" #include "commands/defrem.h" +#include "common/hashfn.h" #include "nodes/makefuncs.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" #include "storage/predicate.h" +#include "utils/lsyscache.h" #include "utils/ruleutils.h" #include "utils/snapmgr.h" #include "utils/syscache.h" @@ -106,7 +109,12 @@ do { \ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap); + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_max); + +static void index_prefetch_tids(IndexScanDesc scan, ScanDirection direction); +static ItemPointer index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_visible); +static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible); /* ---------------------------------------------------------------- @@ -215,18 +223,42 @@ index_insert_cleanup(Relation indexRelation, * index_beginscan - start a scan of an index with amgettuple * * Caller must be holding suitable locks on the heap and the index. + * + * prefetch_max determines if prefetching is requested for this index scan, + * and how far ahead we want to prefetch + * + * Setting prefetch_max to 0 disables prefetching for the index scan. We do + * this for two reasons - for scans on system catalogs, and/or for cases where + * prefetching is expected to be pointless (like IOS). + * + * For system catalogs, we usually either scan by a PK value, or we we expect + * only few rows (or rather we don't know how many rows to expect). Also, we + * need to prevent infinite in the get_tablespace_io_concurrency() call - it + * does an index scan internally. So we simply disable prefetching for system + * catalogs. We could deal with this by picking a conservative static target + * (e.g. effective_io_concurrency, capped to something), but places that are + * performance sensitive likely use syscache anyway, and catalogs tend to be + * very small and hot. So we don't bother. + * + * For IOS, we expect to not need most heap pages (that's the whole point of + * IOS, actually), and prefetching them might lead to a lot of wasted I/O. + * + * XXX Not sure the infinite loop can still happen, now that the target lookup + * moved to callers of index_beginscan. */ IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys) + int nkeys, int norderbys, + int prefetch_max) { IndexScanDesc scan; Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false); + scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, + NULL, false, prefetch_max); /* * Save additional parameters into the scandesc. Everything else was set @@ -256,7 +288,8 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false); + /* No prefetch in bitmap scans, prefetch is done by the heap scan. */ + scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, 0); /* * Save additional parameters into the scandesc. Everything else was set @@ -273,7 +306,8 @@ index_beginscan_bitmap(Relation indexRelation, static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap) + ParallelIndexScanDesc pscan, bool temp_snap, + int prefetch_max) { IndexScanDesc scan; @@ -296,6 +330,31 @@ index_beginscan_internal(Relation indexRelation, /* Initialize information for parallel scan. */ scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; + scan->indexonly = false; + + /* + * With prefetching requested, initialize the prefetcher state. + * + * FIXME This should really be in the IndexScanState, not IndexScanDesc + * (certainly the queues etc). But index_getnext_tid only gets the scan + * descriptor, so how else would we pass it? Seems like a sign of wrong + * layer doing the prefetching. + */ + if ((prefetch_max > 0) && + (io_direct_flags & IO_DIRECT_DATA) == 0) /* no prefetching for direct I/O */ + { + IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); + + prefetcher->queueIndex = 0; + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + + prefetcher->prefetchTarget = 0; + prefetcher->prefetchMaxTarget = prefetch_max; + prefetcher->vmBuffer = InvalidBuffer; + + scan->xs_prefetch = prefetcher; + } return scan; } @@ -332,6 +391,20 @@ index_rescan(IndexScanDesc scan, scan->indexRelation->rd_indam->amrescan(scan, keys, nkeys, orderbys, norderbys); + + /* If we're prefetching for this index, maybe reset some of the state. */ + if (scan->xs_prefetch != NULL) + { + IndexPrefetch prefetcher = scan->xs_prefetch; + + prefetcher->queueStart = 0; + prefetcher->queueEnd = 0; + prefetcher->queueIndex = 0; + prefetcher->prefetchDone = false; + + /* restart the incremental ramp-up */ + prefetcher->prefetchTarget = 0; + } } /* ---------------- @@ -360,6 +433,23 @@ index_endscan(IndexScanDesc scan) if (scan->xs_temp_snap) UnregisterSnapshot(scan->xs_snapshot); + /* + * If prefetching was enabled for this scan, log prefetch stats. + * + * FIXME This should really go to EXPLAIN ANALYZE instead. + */ + if (scan->xs_prefetch) + { + IndexPrefetch prefetch = scan->xs_prefetch; + + elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, + prefetch->countAll, + prefetch->countPrefetch, + prefetch->countPrefetch * 100.0 / prefetch->countAll, + prefetch->countSkipCached, + prefetch->countSkipSequential); + } + /* Release the scan data structure itself */ IndexScanEnd(scan); } @@ -505,7 +595,8 @@ index_parallelrescan(IndexScanDesc scan) */ IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, - int norderbys, ParallelIndexScanDesc pscan) + int norderbys, ParallelIndexScanDesc pscan, + int prefetch_max) { Snapshot snapshot; IndexScanDesc scan; @@ -514,7 +605,7 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, snapshot = RestoreSnapshot(pscan->ps_snapshot_data); RegisterSnapshot(snapshot); scan = index_beginscan_internal(indexrel, nkeys, norderbys, snapshot, - pscan, true); + pscan, true, prefetch_max); /* * Save additional parameters into the scandesc. Everything else was set @@ -536,8 +627,8 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, * or NULL if no more matching tuples exist. * ---------------- */ -ItemPointer -index_getnext_tid(IndexScanDesc scan, ScanDirection direction) +static ItemPointer +index_getnext_tid_internal(IndexScanDesc scan, ScanDirection direction) { bool found; @@ -640,12 +731,16 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot * { for (;;) { + /* Do prefetching (if requested/enabled). */ + index_prefetch_tids(scan, direction); + if (!scan->xs_heap_continue) { - ItemPointer tid; + ItemPointer tid; + bool all_visible; /* Time to fetch the next TID from the index */ - tid = index_getnext_tid(scan, direction); + tid = index_prefetch_get_tid(scan, direction, &all_visible); /* If we're out of index entries, we're done */ if (tid == NULL) @@ -1003,3 +1098,531 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, return build_local_reloptions(&relopts, attoptions, validate); } + +/* + * index_prefetch_is_sequential + * Track the block number and check if the I/O pattern is sequential, + * or if the same block was just prefetched. + * + * Prefetching is cheap, but for some access patterns the benefits are small + * compared to the extra overhead. In particular, for sequential access the + * read-ahead performed by the OS is very effective/efficient. Doing more + * prefetching is just increasing the costs. + * + * This tries to identify simple sequential patterns, so that we can skip + * the prefetching request. This is implemented by having a small queue + * of block numbers, and checking it before prefetching another block. + * + * We look at the preceding PREFETCH_SEQ_PATTERN_BLOCKS blocks, and see if + * they are sequential. We also check if the block is the same as the last + * request (which is not sequential). + * + * Note that the main prefetch queue is not really useful for this, as it + * stores TIDs while we care about block numbers. Consider a sorted table, + * with a perfectly sequential pattern when accessed through an index. Each + * heap page may have dozens of TIDs, but we need to check block numbers. + * We could keep enough TIDs to cover enough blocks, but then we also need + * to walk those when checking the pattern (in hot path). + * + * So instead, we maintain a small separate queue of block numbers, and we use + * this instead. + * + * Returns true if the block is in a sequential pattern (and so should not be + * prefetched), or false (not sequential, should be prefetched). + * + * XXX The name is a bit misleading, as it also adds the block number to the + * block queue and checks if the block is the same as the last one (which + * does not require a sequential pattern). + */ +static bool +index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) +{ + int idx; + + /* + * If the block queue is empty, just store the block and we're done (it's + * neither a sequential pattern, neither recently prefetched block). + */ + if (prefetch->blockIndex == 0) + { + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + return false; + } + + /* + * Check if it's the same as the immediately preceding block. We don't + * want to prefetch the same block over and over (which would happen for + * well correlated indexes). + * + * In principle we could rely on index_prefetch_add_cache doing this using + * the full cache, but this check is much cheaper and we need to look at + * the preceding block anyway, so we just do it. + * + * XXX Notice we haven't added the block to the block queue yet, and there + * is a preceding block (i.e. blockIndex-1 is valid). + */ + if (prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex - 1)] == block) + return true; + + /* + * Add the block number to the queue. + * + * We do this before checking if the pattern, because we want to know + * about the block even if we end up skipping the prefetch. Otherwise we'd + * not be able to detect longer sequential pattens - we'd skip one block + * but then fail to skip the next couple blocks even in a perfect + * sequential pattern. This ocillation might even prevent the OS + * read-ahead from kicking in. + */ + prefetch->blockItems[PREFETCH_BLOCK_INDEX(prefetch->blockIndex)] = block; + prefetch->blockIndex++; + + /* + * Check if the last couple blocks are in a sequential pattern. We look + * for a sequential pattern of PREFETCH_SEQ_PATTERN_BLOCKS (4 by default), + * so we look for patterns of 5 pages (40kB) including the new block. + * + * XXX Perhaps this should be tied to effective_io_concurrency somehow? + * + * XXX Could it be harmful that we read the queue backwards? Maybe memory + * prefetching works better for the forward direction? + */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) + { + /* + * Are there enough requests to confirm a sequential pattern? We only + * consider something to be sequential after finding a sequence of + * PREFETCH_SEQ_PATTERN_BLOCKS blocks. + * + * FIXME Better to move this outside the loop. + */ + if (prefetch->blockIndex < i) + return false; + + /* + * Calculate index of the earlier block (we need to do -1 as we + * already incremented the index when adding the new block to the + * queue). + */ + idx = PREFETCH_BLOCK_INDEX(prefetch->blockIndex - i - 1); + + /* + * For a sequential pattern, blocks "k" step ago needs to have block + * number by "k" smaller compared to the current block. + */ + if (prefetch->blockItems[idx] != (block - i)) + return false; + } + + return true; +} + +/* + * index_prefetch_add_cache + * Add a block to the cache, check if it was recently prefetched. + * + * We don't want to prefetch blocks that we already prefetched recently. It's + * cheap but not free, and the overhead may have measurable impact. + * + * This check needs to be very cheap, even with fairly large caches (hundreds + * of entries, see PREFETCH_CACHE_SIZE). + * + * A simple queue would allow expiring the requests, but checking if it + * contains a particular block prefetched would be expensive (linear search). + * Another option would be a simple hash table, which has fast lookup but + * does not allow expiring entries cheaply. + * + * The cache does not need to be perfect, we can accept false + * positives/negatives, as long as the rate is reasonably low. We also need + * to expire entries, so that only "recent" requests are remembered. + * + * We use a hybrid cache that is organized as many small LRU caches. Each + * block is mapped to a particular LRU by hashing (so it's a bit like a + * hash table). The LRU caches are tiny (e.g. 8 entries), and the expiration + * happens at the level of a single LRU (by tracking only the 8 most recent requests). + * + * This allows quick searches and expiration, but with false negatives (when a + * particular LRU has too many collisions, we may evict entries that are more + * recent than some other LRU). + * + * For example, imagine 128 LRU caches, each with 8 entries - that's 1024 + * prefetch request in total (these are the default parameters.) + * + * The recency is determined using a prefetch counter, incremented every + * time we end up prefetching a block. The counter is uint64, so it should + * not wrap (125 zebibytes, would take ~4 million years at 1GB/s). + * + * To check if a block was prefetched recently, we calculate hash(block), + * and then linearly search if the tiny LRU has entry for the same block + * and request less than PREFETCH_CACHE_SIZE ago. + * + * At the same time, we either update the entry (for the queried block) if + * found, or replace the oldest/empty entry. + * + * If the block was not recently prefetched (i.e. we want to prefetch it), + * we increment the counter. + * + * Returns true if the block was recently prefetched (and thus we don't + * need to prefetch it again), or false (should do a prefetch). + * + * XXX It's a bit confusing these return values are inverse compared to + * what index_prefetch_is_sequential does. + */ +static bool +index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) +{ + PrefetchCacheEntry *entry; + + /* map the block number the the LRU */ + int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; + + /* age/index of the oldest entry in the LRU, to maybe use */ + uint64 oldestRequest = PG_UINT64_MAX; + int oldestIndex = -1; + + /* + * First add the block to the (tiny) top-level LRU cache and see if it's + * part of a sequential pattern. In this case we just ignore the block and + * don't prefetch it - we expect read-ahead to do a better job. + * + * XXX Maybe we should still add the block to the hybrid cache, in case we + * happen to access it later? That might help if we first scan a lot of + * the table sequentially, and then randomly. Not sure that's very likely + * with index access, though. + */ + if (index_prefetch_is_sequential(prefetch, block)) + { + prefetch->countSkipSequential++; + return true; + } + + /* + * See if we recently prefetched this block - we simply scan the LRU + * linearly. While doing that, we also track the oldest entry, so that we + * know where to put the block if we don't find a matching entry. + */ + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) + { + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; + + /* Is this the oldest prefetch request in this LRU? */ + if (entry->request < oldestRequest) + { + oldestRequest = entry->request; + oldestIndex = i; + } + + /* + * If the entry is unused (identified by request being set to 0), + * we're done. Notice the field is uint64, so empty entry is + * guaranteed to be the oldest one. + */ + if (entry->request == 0) + continue; + + /* Is this entry for the same block as the current request? */ + if (entry->block == block) + { + bool prefetched; + + /* + * Is the old request sufficiently recent? If yes, we treat the + * block as already prefetched. + * + * XXX We do add the cache size to the request in order not to + * have issues with uint64 underflows. + */ + prefetched = ((entry->request + PREFETCH_CACHE_SIZE) >= prefetch->prefetchReqNumber); + + /* Update the request number. */ + entry->request = ++prefetch->prefetchReqNumber; + + prefetch->countSkipCached += (prefetched) ? 1 : 0; + + return prefetched; + } + } + + /* + * We didn't find the block in the LRU, so store it either in an empty + * entry, or in the "oldest" prefetch request in this LRU. + */ + Assert((oldestIndex >= 0) && (oldestIndex < PREFETCH_LRU_SIZE)); + + /* FIXME do a nice macro */ + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + oldestIndex]; + + entry->block = block; + entry->request = ++prefetch->prefetchReqNumber; + + /* not in the prefetch cache */ + return false; +} + +/* + * index_prefetch + * Prefetch the TID, unless it's sequential or recently prefetched. + * + * XXX Some ideas how to auto-tune the prefetching, so that unnecessary + * prefetching does not cause significant regressions (e.g. for nestloop + * with inner index scan). We could track number of rescans and number of + * items (TIDs) actually returned from the scan. Then we could calculate + * rows / rescan and use that to clamp prefetch target. + * + * That'd help with cases when a scan matches only very few rows, far less + * than the prefetchTarget, because the unnecessary prefetches are wasted + * I/O. Imagine a LIMIT on top of index scan, or something like that. + * + * Another option is to use the planner estimates - we know how many rows we're + * expecting to fetch (on average, assuming the estimates are reasonably + * accurate), so why not to use that? + * + * Of course, we could/should combine these two approaches. + * + * XXX The prefetching may interfere with the patch allowing us to evaluate + * conditions on the index tuple, in which case we may not need the heap + * tuple. Maybe if there's such filter, we should prefetch only pages that + * are not all-visible (and the same idea would also work for IOS), but + * it also makes the indexing a bit "aware" of the visibility stuff (which + * seems a somewhat wrong). Also, maybe we should consider the filter selectivity + * (if the index-only filter is expected to eliminate only few rows, then + * the vm check is pointless). Maybe this could/should be auto-tuning too, + * i.e. we could track how many heap tuples were needed after all, and then + * we would consider this when deciding whether to prefetch all-visible + * pages or not (matters only for regular index scans, not IOS). + * + * XXX Maybe we could/should also prefetch the next index block, e.g. stored + * in BTScanPosData.nextPage. + * + * XXX Could we tune the cache size based on execution statistics? We have + * a cache of limited size (PREFETCH_CACHE_SIZE = 1024 by default), but + * how do we know it's the right size? Ideally, we'd have a cache large + * enough to track actually cached blocks. If the OS caches 10240 pages, + * then we may do 90% of prefetch requests unnecessarily. Or maybe there's + * a lot of contention, blocks are evicted quickly, and 90% of the blocks + * in the cache are not actually cached anymore? But we do have a concept + * of sequential request ID (PrefetchCacheEntry->request), which gives us + * information about "age" of the last prefetch. Now it's used only when + * evicting entries (to keep the more recent one), but maybe we could also + * use it when deciding if the page is cached. Right now any block that's + * in the cache is considered cached and not prefetched, but maybe we could + * have "max age", and tune it based on feedback from reading the blocks + * later. For example, if we find the block in cache and decide not to + * prefetch it, but then later find we have to do I/O, it means our cache + * is too large. And we could "reduce" the maximum age (measured from the + * current prefetchReqNumber value), so that only more recent blocks would + * be considered cached. Not sure about the opposite direction, where we + * decide to prefetch a block - AFAIK we don't have a way to determine if + * I/O was needed or not in this case (so we can't increase the max age). + * But maybe we could di that somehow speculatively, i.e. increase the + * value once in a while, and see what happens. + */ +static void +index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible) +{ + IndexPrefetch prefetch = scan->xs_prefetch; + BlockNumber block; + + /* by default not all visible (or we didn't check) */ + *all_visible = false; + + /* + * No heap relation means bitmap index scan, which does prefetching at the + * bitmap heap scan, so no prefetch here (we can't do it anyway, without + * the heap) + * + * XXX But in this case we should have prefetchMaxTarget=0, because in + * index_bebinscan_bitmap() we disable prefetching. So maybe we should + * just check that. + */ + if (!prefetch) + return; + + /* + * If we got here, prefetching is enabled and it's a node that supports + * prefetching (i.e. it can't be a bitmap index scan). + */ + Assert(scan->heapRelation); + + block = ItemPointerGetBlockNumber(tid); + + /* + * When prefetching for IOS, we want to only prefetch pages that are not + * marked as all-visible (because not fetching all-visible pages is the + * point of IOS). + * + * XXX This is not great, because it releases the VM buffer for each TID + * we consider to prefetch. We should reuse that somehow, similar to the + * actual IOS code. Ideally, we should use the same ioss_VMBuffer (if + * we can propagate it here). Or at least do it for a bulk of prefetches, + * although that's not very useful - after the ramp-up we will prefetch + * the pages one by one anyway. + * + * XXX Ideally we'd also propagate this to the executor, so that the + * nodeIndexonlyscan.c doesn't need to repeat the same VM check (which + * is measurable). But the index_getnext_tid() is not really well + * suited for that, so the API needs a change.s + */ + if (skip_all_visible) + { + *all_visible = VM_ALL_VISIBLE(scan->heapRelation, + block, + &prefetch->vmBuffer); + + if (*all_visible) + return; + } + + /* + * Do not prefetch the same block over and over again, + * + * This happens e.g. for clustered or naturally correlated indexes (fkey + * to a sequence ID). It's not expensive (the block is in page cache + * already, so no I/O), but it's not free either. + */ + if (!index_prefetch_add_cache(prefetch, block)) + { + prefetch->countPrefetch++; + + PrefetchBuffer(scan->heapRelation, MAIN_FORKNUM, block); + pgBufferUsage.blks_prefetches++; + } + + prefetch->countAll++; +} + +/* ---------------- + * index_getnext_tid - get the next TID from a scan + * + * The result is the next TID satisfying the scan keys, + * or NULL if no more matching tuples exist. + * + * FIXME not sure this handles xs_heapfetch correctly. + * ---------------- + */ +ItemPointer +index_getnext_tid(IndexScanDesc scan, ScanDirection direction) +{ + bool all_visible; /* ignored */ + + /* Do prefetching (if requested/enabled). */ + index_prefetch_tids(scan, direction); + + /* Read the TID from the queue (or directly from the index). */ + return index_prefetch_get_tid(scan, direction, &all_visible); +} + +ItemPointer +index_getnext_tid_vm(IndexScanDesc scan, ScanDirection direction, bool *all_visible) +{ + /* Do prefetching (if requested/enabled). */ + index_prefetch_tids(scan, direction); + + /* Read the TID from the queue (or directly from the index). */ + return index_prefetch_get_tid(scan, direction, all_visible); +} + +static void +index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) +{ + /* for convenience */ + IndexPrefetch prefetch = scan->xs_prefetch; + + /* + * If the prefetching is still active (i.e. enabled and we still + * haven't finished reading TIDs from the scan), read enough TIDs into + * the queue until we hit the current target. + */ + if (PREFETCH_ACTIVE(prefetch)) + { + /* + * Ramp up the prefetch distance incrementally. + * + * Intentionally done as first, before reading the TIDs into the + * queue, so that there's always at least one item. Otherwise we + * might get into a situation where we start with target=0 and no + * TIDs loaded. + */ + prefetch->prefetchTarget = Min(prefetch->prefetchTarget + 1, + prefetch->prefetchMaxTarget); + + /* + * Now read TIDs from the index until the queue is full (with + * respect to the current prefetch target). + */ + while (!PREFETCH_FULL(prefetch)) + { + ItemPointer tid; + bool all_visible; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid_internal(scan, direction); + + /* + * If we're out of index entries, we're done (and we mark the + * the prefetcher as inactive). + */ + if (tid == NULL) + { + prefetch->prefetchDone = true; + break; + } + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + + /* + * Issue the actuall prefetch requests for the new TID. + * + * XXX index_getnext_tid_prefetch is only called for IOS (for now), + * so skip prefetching of all-visible pages. + */ + index_prefetch(scan, tid, scan->indexonly, &all_visible); + + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)].tid = *tid; + prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)].all_visible = all_visible; + prefetch->queueEnd++; + } + } +} + +static ItemPointer +index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_visible) +{ + /* for convenience */ + IndexPrefetch prefetch = scan->xs_prefetch; + + /* + * With prefetching enabled (even if we already finished reading + * all TIDs from the index scan), we need to return a TID from the + * queue. Otherwise, we just get the next TID from the scan + * directly. + */ + if (PREFETCH_ENABLED(prefetch)) + { + /* Did we reach the end of the scan and the queue is empty? */ + if (PREFETCH_DONE(prefetch)) + return NULL; + + scan->xs_heaptid = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)].tid; + *all_visible = prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueIndex)].all_visible; + prefetch->queueIndex++; + } + else /* not prefetching, just do the regular work */ + { + ItemPointer tid; + + /* Time to fetch the next TID from the index */ + tid = index_getnext_tid_internal(scan, direction); + *all_visible = false; + + /* If we're out of index entries, we're done */ + if (tid == NULL) + return NULL; + + Assert(ItemPointerEquals(tid, &scan->xs_heaptid)); + } + + /* Return the TID of the tuple we found. */ + return &scan->xs_heaptid; +} diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index f1d71bc54e8..6810996edfd 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3568,6 +3568,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) !INSTR_TIME_IS_ZERO(usage->local_blk_write_time)); bool has_temp_timing = (!INSTR_TIME_IS_ZERO(usage->temp_blk_read_time) || !INSTR_TIME_IS_ZERO(usage->temp_blk_write_time)); + bool has_prefetches = (usage->blks_prefetches > 0); bool show_planning = (planning && (has_shared || has_local || has_temp || has_shared_timing || @@ -3679,6 +3680,23 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage, bool planning) appendStringInfoChar(es->str, '\n'); } + /* As above, show only positive counter values. */ + if (has_prefetches) + { + ExplainIndentText(es); + appendStringInfoString(es->str, "Prefetches:"); + + if (usage->blks_prefetches > 0) + appendStringInfo(es->str, " blocks=%lld", + (long long) usage->blks_prefetches); + + if (usage->blks_prefetch_rounds > 0) + appendStringInfo(es->str, " rounds=%lld", + (long long) usage->blks_prefetch_rounds); + + appendStringInfoChar(es->str, '\n'); + } + if (show_planning) es->indent--; } diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 2fa2118f3c2..15fa3211667 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -770,11 +770,15 @@ check_exclusion_or_unique_constraint(Relation heap, Relation index, /* * May have to restart scan from this point if a potential conflict is * found. + * + * XXX Should this do index prefetch? Probably not worth it for unique + * constraints, I guess? Otherwise we should calculate prefetch_target + * just like in nodeIndexscan etc. */ retry: conflict = false; found_self = false; - index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0); + index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0); index_rescan(index_scan, scankeys, indnkeyatts, NULL, 0); while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot)) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 81f27042bc4..91676ccff95 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -204,8 +204,13 @@ RelationFindReplTupleByIndex(Relation rel, Oid idxoid, /* Build scan key. */ skey_attoff = build_replindex_scan_key(skey, rel, idxrel, searchslot); - /* Start an index scan. */ - scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0); + /* Start an index scan. + * + * XXX Should this do index prefetching? We're looking for a single tuple, + * probably using a PK / UNIQUE index, so does not seem worth it. If we + * reconsider this, calclate prefetch_target like in nodeIndexscan. + */ + scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0); retry: found = false; diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c index c383f34c066..0011d9f679c 100644 --- a/src/backend/executor/instrument.c +++ b/src/backend/executor/instrument.c @@ -235,6 +235,8 @@ BufferUsageAdd(BufferUsage *dst, const BufferUsage *add) dst->local_blks_written += add->local_blks_written; dst->temp_blks_read += add->temp_blks_read; dst->temp_blks_written += add->temp_blks_written; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds; + dst->blks_prefetches += add->blks_prefetches; INSTR_TIME_ADD(dst->shared_blk_read_time, add->shared_blk_read_time); INSTR_TIME_ADD(dst->shared_blk_write_time, add->shared_blk_write_time); INSTR_TIME_ADD(dst->local_blk_read_time, add->local_blk_read_time); @@ -259,6 +261,8 @@ BufferUsageAccumDiff(BufferUsage *dst, dst->local_blks_written += add->local_blks_written - sub->local_blks_written; dst->temp_blks_read += add->temp_blks_read - sub->temp_blks_read; dst->temp_blks_written += add->temp_blks_written - sub->temp_blks_written; + dst->blks_prefetches += add->blks_prefetches - sub->blks_prefetches; + dst->blks_prefetch_rounds += add->blks_prefetch_rounds - sub->blks_prefetch_rounds; INSTR_TIME_ACCUM_DIFF(dst->shared_blk_read_time, add->shared_blk_read_time, sub->shared_blk_read_time); INSTR_TIME_ACCUM_DIFF(dst->shared_blk_write_time, diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index f1db35665c8..b6660c10a63 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -43,7 +43,7 @@ #include "storage/predicate.h" #include "utils/memutils.h" #include "utils/rel.h" - +#include "utils/spccache.h" static TupleTableSlot *IndexOnlyNext(IndexOnlyScanState *node); static void StoreIndexTuple(TupleTableSlot *slot, IndexTuple itup, @@ -65,6 +65,8 @@ IndexOnlyNext(IndexOnlyScanState *node) IndexScanDesc scandesc; TupleTableSlot *slot; ItemPointer tid; + Relation heapRel = node->ss.ss_currentRelation; + bool all_visible; /* * extract necessary information from index scan node @@ -83,16 +85,47 @@ IndexOnlyNext(IndexOnlyScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index only scan is not parallel, or if we're * serially executing an index only scan that was planned to be * parallel. + * + * XXX Maybe we should enable prefetching, but prefetch only pages that + * are not all-visible (but checking that from the index code seems like + * a violation of layering etc). + * + * XXX This might lead to IOS being slower than plain index scan, if the + * table has a lot of pages that need recheck. */ scandesc = index_beginscan(node->ss.ss_currentRelation, node->ioss_RelationDesc, estate->es_snapshot, node->ioss_NumScanKeys, - node->ioss_NumOrderByKeys); + node->ioss_NumOrderByKeys, + prefetch_max); + + /* + * Remember this is index-only scan, because of prefetching. Not the most + * elegant way to pass this info. + */ + scandesc->indexonly = true; node->ioss_ScanDesc = scandesc; @@ -116,7 +149,7 @@ IndexOnlyNext(IndexOnlyScanState *node) /* * OK, now that we have what we need, fetch the next tuple. */ - while ((tid = index_getnext_tid(scandesc, direction)) != NULL) + while ((tid = index_getnext_tid_vm(scandesc, direction, &all_visible)) != NULL) { bool tuple_from_heap = false; @@ -155,8 +188,11 @@ IndexOnlyNext(IndexOnlyScanState *node) * * It's worth going through this complexity to avoid needing to lock * the VM buffer, which could cause significant contention. + * + * XXX Skip if we already know the page is all visible from prefetcher. */ - if (!VM_ALL_VISIBLE(scandesc->heapRelation, + if (!all_visible && + !VM_ALL_VISIBLE(scandesc->heapRelation, ItemPointerGetBlockNumber(tid), &node->ioss_VMBuffer)) { @@ -380,6 +416,18 @@ ExecEndIndexOnlyScan(IndexOnlyScanState *node) node->ioss_VMBuffer = InvalidBuffer; } + /* Release VM buffer pin from prefetcher, if any. */ + if (indexScanDesc && indexScanDesc->xs_prefetch) + { + IndexPrefetch indexPrefetch = indexScanDesc->xs_prefetch; + + if (indexPrefetch->vmBuffer != InvalidBuffer) + { + ReleaseBuffer(indexPrefetch->vmBuffer); + indexPrefetch->vmBuffer = InvalidBuffer; + } + } + /* * close the index relation (no-op if we didn't open it) */ @@ -646,6 +694,24 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->ioss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -658,7 +724,8 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -696,6 +763,23 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->ioss_ScanDesc = @@ -703,7 +787,8 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 14b9c00217a..a5f5394ef49 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -43,6 +43,7 @@ #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "utils/spccache.h" /* * When an ordering operator is used, tuples fetched from the index that @@ -85,6 +86,7 @@ IndexNext(IndexScanState *node) ScanDirection direction; IndexScanDesc scandesc; TupleTableSlot *slot; + Relation heapRel = node->ss.ss_currentRelation; /* * extract necessary information from index scan node @@ -103,6 +105,21 @@ IndexNext(IndexScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index scan. This + * is essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -111,7 +128,8 @@ IndexNext(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_max); node->iss_ScanDesc = scandesc; @@ -177,6 +195,7 @@ IndexNextWithReorder(IndexScanState *node) Datum *lastfetched_vals; bool *lastfetched_nulls; int cmp; + Relation heapRel = node->ss.ss_currentRelation; estate = node->ss.ps.state; @@ -198,6 +217,21 @@ IndexNextWithReorder(IndexScanState *node) if (scandesc == NULL) { + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); + /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -206,7 +240,8 @@ IndexNextWithReorder(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys); + node->iss_NumOrderByKeys, + prefetch_max); node->iss_ScanDesc = scandesc; @@ -1662,6 +1697,24 @@ ExecIndexScanInitializeDSM(IndexScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -1674,7 +1727,8 @@ ExecIndexScanInitializeDSM(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); /* * If no run-time keys to calculate or they are ready, go ahead and pass @@ -1710,6 +1764,23 @@ ExecIndexScanInitializeWorker(IndexScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; + Relation heapRel = node->ss.ss_currentRelation; + int prefetch_max; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->iss_ScanDesc = @@ -1717,7 +1788,8 @@ ExecIndexScanInitializeWorker(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan); + piscan, + prefetch_max); /* * If no run-time keys to calculate or they are ready, go ahead and pass diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index e11d022827a..8b662d371dd 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -6289,9 +6289,10 @@ get_actual_variable_endpoint(Relation heapRel, InitNonVacuumableSnapshot(SnapshotNonVacuumable, GlobalVisTestFor(heapRel)); + /* XXX Maybe should do prefetching using the default prefetch parameters? */ index_scan = index_beginscan(heapRel, indexRel, &SnapshotNonVacuumable, - 1, 0); + 1, 0, 0); /* Set it up for index-only scan */ index_scan->xs_want_itup = true; index_rescan(index_scan, scankeys, 1, NULL, 0); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 80dc8d54066..f6882f644d2 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -17,6 +17,7 @@ #include "access/sdir.h" #include "access/skey.h" #include "nodes/tidbitmap.h" +#include "storage/bufmgr.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -154,7 +155,8 @@ extern void index_insert_cleanup(Relation indexRelation, extern IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys); + int nkeys, int norderbys, + int prefetch_max); extern IndexScanDesc index_beginscan_bitmap(Relation indexRelation, Snapshot snapshot, int nkeys); @@ -171,9 +173,13 @@ extern void index_parallelscan_initialize(Relation heapRelation, extern void index_parallelrescan(IndexScanDesc scan); extern IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, - ParallelIndexScanDesc pscan); + ParallelIndexScanDesc pscan, + int prefetch_max); extern ItemPointer index_getnext_tid(IndexScanDesc scan, ScanDirection direction); +extern ItemPointer index_getnext_tid_vm(IndexScanDesc scan, + ScanDirection direction, + bool *all_visible); struct TupleTableSlot; extern bool index_fetch_heap(IndexScanDesc scan, struct TupleTableSlot *slot); extern bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, @@ -232,4 +238,104 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); +/* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. Doesn't need to be perfectly accurate, but we + * aim to make false positives/negatives reasonably low. + */ +typedef struct PrefetchCacheEntry { + BlockNumber block; + uint64 request; +} PrefetchCacheEntry; + +/* + * Size of the cache of recently prefetched blocks - shouldn't be too + * small or too large. 1024 seems about right, it covers ~8MB of data. + * It's somewhat arbitrary, there's no particular formula saying it + * should not be higher/lower. + * + * The cache is structured as an array of small LRU caches, so the total + * size needs to be a multiple of LRU size. The LRU should be tiny to + * keep linear search cheap enough. + * + * XXX Maybe we could consider effective_cache_size or something? + */ +#define PREFETCH_LRU_SIZE 8 +#define PREFETCH_LRU_COUNT 128 +#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) + +/* + * Used to detect sequential patterns (and disable prefetching). + */ +#define PREFETCH_QUEUE_HISTORY 8 +#define PREFETCH_SEQ_PATTERN_BLOCKS 4 + +typedef struct PrefetchEntry +{ + ItemPointerData tid; + bool all_visible; +} PrefetchEntry; + +typedef struct IndexPrefetchData +{ + /* + * XXX We need to disable this in some cases (e.g. when using index-only + * scans, we don't want to prefetch pages). Or maybe we should prefetch + * only pages that are not all-visible, that'd be even better. + */ + int prefetchTarget; /* how far we should be prefetching */ + int prefetchMaxTarget; /* maximum prefetching distance */ + int prefetchReset; /* reset to this distance on rescan */ + bool prefetchDone; /* did we get all TIDs from the index? */ + + /* runtime statistics */ + uint64 countAll; /* all prefetch requests */ + uint64 countPrefetch; /* actual prefetches */ + uint64 countSkipSequential; + uint64 countSkipCached; + + /* used when prefetching index-only scans */ + Buffer vmBuffer; + + /* + * Queue of TIDs to prefetch. + * + * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler + * than dynamically adjusting for custom values. + */ + PrefetchEntry queueItems[MAX_IO_CONCURRENCY]; + uint64 queueIndex; /* next TID to prefetch */ + uint64 queueStart; /* first valid TID in queue */ + uint64 queueEnd; /* first invalid (empty) TID in queue */ + + /* + * A couple of last prefetched blocks, used to check for certain access + * pattern and skip prefetching - e.g. for sequential access). + * + * XXX Separate from the main queue, because we only want to compare the + * block numbers, not the whole TID. In sequential access it's likely we + * read many items from each page, and we don't want to check many items + * (as that is much more expensive). + */ + BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; + uint64 blockIndex; /* index in the block (points to the first + * empty entry)*/ + + /* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. + */ + uint64 prefetchReqNumber; + PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; + +} IndexPrefetchData; + +#define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) +#define PREFETCH_QUEUE_EMPTY(p) ((p)->queueEnd == (p)->queueIndex) +#define PREFETCH_ENABLED(p) ((p) && ((p)->prefetchMaxTarget > 0)) +#define PREFETCH_FULL(p) ((p)->queueEnd - (p)->queueIndex == (p)->prefetchTarget) +#define PREFETCH_DONE(p) ((p) && ((p)->prefetchDone && PREFETCH_QUEUE_EMPTY(p))) +#define PREFETCH_ACTIVE(p) (PREFETCH_ENABLED(p) && !(p)->prefetchDone) +#define PREFETCH_BLOCK_INDEX(v) ((v) % PREFETCH_QUEUE_HISTORY) + #endif /* GENAM_H */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index d03360eac04..d5903492c6e 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -106,6 +106,12 @@ typedef struct IndexFetchTableData Relation rel; } IndexFetchTableData; +/* + * Forward declarations, defined in genam.h. + */ +typedef struct IndexPrefetchData IndexPrefetchData; +typedef struct IndexPrefetchData *IndexPrefetch; + /* * We use the same IndexScanDescData structure for both amgettuple-based * and amgetbitmap-based index scans. Some fields are only relevant in @@ -129,6 +135,7 @@ typedef struct IndexScanDescData bool ignore_killed_tuples; /* do not return killed entries */ bool xactStartedInRecovery; /* prevents killing/seeing killed * tuples */ + bool indexonly; /* is this index-only scan? */ /* index access method's private state */ void *opaque; /* access-method-specific info */ @@ -162,6 +169,9 @@ typedef struct IndexScanDescData bool *xs_orderbynulls; bool xs_recheckorderby; + /* prefetching state (or NULL if disabled for this scan) */ + IndexPrefetchData *xs_prefetch; + /* parallel index scan information, in shared memory */ struct ParallelIndexScanDescData *parallel_scan; } IndexScanDescData; diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index d5d69941c52..f53fb4a1e51 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -33,6 +33,8 @@ typedef struct BufferUsage int64 local_blks_written; /* # of local disk blocks written */ int64 temp_blks_read; /* # of temp blocks read */ int64 temp_blks_written; /* # of temp blocks written */ + int64 blks_prefetch_rounds; /* # of prefetch rounds */ + int64 blks_prefetches; /* # of buffers prefetched */ instr_time shared_blk_read_time; /* time spent reading shared blocks */ instr_time shared_blk_write_time; /* time spent writing shared blocks */ instr_time local_blk_read_time; /* time spent reading local blocks */ -- 2.41.0 [text/x-patch] v20231209-0002-reworks.patch (49.6K, 3-v20231209-0002-reworks.patch) download | inline diff: From b0c3e346bf41c6c7d14502814bb0ee327ae68169 Mon Sep 17 00:00:00 2001 From: Tomas Vondra <[email protected]> Date: Sat, 9 Dec 2023 00:21:30 +0100 Subject: [PATCH v20231209 2/2] reworks --- src/backend/access/heap/heapam_handler.c | 14 +- src/backend/access/index/genam.c | 35 +--- src/backend/access/index/indexam.c | 152 ++++------------ src/backend/executor/execIndexing.c | 12 +- src/backend/executor/execReplication.c | 18 +- src/backend/executor/nodeIndexonlyscan.c | 166 ++++++++--------- src/backend/executor/nodeIndexscan.c | 149 ++++++++-------- src/backend/utils/adt/selfuncs.c | 5 +- src/include/access/genam.h | 217 ++++++++++++----------- src/include/access/relscan.h | 10 -- src/include/nodes/execnodes.h | 4 + 11 files changed, 332 insertions(+), 450 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 89474078951..26d3ec20b63 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -44,7 +44,6 @@ #include "storage/smgr.h" #include "utils/builtins.h" #include "utils/rel.h" -#include "utils/spccache.h" static void reform_and_rewrite_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, @@ -748,14 +747,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, PROGRESS_CLUSTER_INDEX_RELID }; int64 ci_val[2]; - int prefetch_max; - - /* - * Get the prefetch target for the old tablespace (which is what we'll - * read using the index). We'll use it as a reset value too, although - * there should be no rescans for CLUSTER etc. - */ - prefetch_max = get_tablespace_io_concurrency(OldHeap->rd_rel->reltablespace); /* Set phase and OIDOldIndex to columns */ ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; @@ -764,8 +755,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, tableScan = NULL; heapScan = NULL; - indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0, - prefetch_max); + indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0); index_rescan(indexScan, NULL, 0, NULL, 0); } else @@ -802,7 +792,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (indexScan != NULL) { - if (!index_getnext_slot(indexScan, ForwardScanDirection, slot)) + if (!index_getnext_slot(indexScan, ForwardScanDirection, slot, NULL)) break; /* Since we used no scan keys, should never need to recheck */ diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index d45a209ee3a..72e7c9f206c 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -126,9 +126,6 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys) scan->xs_hitup = NULL; scan->xs_hitupdesc = NULL; - /* Information used for asynchronous prefetching during index scans. */ - scan->xs_prefetch = NULL; - return scan; } @@ -443,20 +440,8 @@ systable_beginscan(Relation heapRelation, elog(ERROR, "column is not in index"); } - /* - * We don't do any prefetching on system catalogs, for two main reasons. - * - * Firstly, we usually do PK lookups, which makes prefetching pointles, - * or we often don't know how many rows to expect (and the numbers tend - * to be fairly low). So it's not clear it'd help. Furthermore, places - * that are sensitive tend to use syscache anyway. - * - * Secondly, we can't call get_tablespace_io_concurrency() because that - * does a sysscan internally, so it might lead to a cycle. We could use - * use effective_io_concurrency, but it doesn't seem worth it. - */ sysscan->iscan = index_beginscan(heapRelation, irel, - snapshot, nkeys, 0, 0); + snapshot, nkeys, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; } @@ -524,7 +509,7 @@ systable_getnext(SysScanDesc sysscan) if (sysscan->irel) { - if (index_getnext_slot(sysscan->iscan, ForwardScanDirection, sysscan->slot)) + if (index_getnext_slot(sysscan->iscan, ForwardScanDirection, sysscan->slot, NULL)) { bool shouldFree; @@ -711,20 +696,8 @@ systable_beginscan_ordered(Relation heapRelation, elog(ERROR, "column is not in index"); } - /* - * We don't do any prefetching on system catalogs, for two main reasons. - * - * Firstly, we usually do PK lookups, which makes prefetching pointles, - * or we often don't know how many rows to expect (and the numbers tend - * to be fairly low). So it's not clear it'd help. Furthermore, places - * that are sensitive tend to use syscache anyway. - * - * Secondly, we can't call get_tablespace_io_concurrency() because that - * does a sysscan internally, so it might lead to a cycle. We could use - * use effective_io_concurrency, but it doesn't seem worth it. - */ sysscan->iscan = index_beginscan(heapRelation, indexRelation, - snapshot, nkeys, 0, 0); + snapshot, nkeys, 0); index_rescan(sysscan->iscan, key, nkeys, NULL, 0); sysscan->scan = NULL; @@ -740,7 +713,7 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction) HeapTuple htup = NULL; Assert(sysscan->irel); - if (index_getnext_slot(sysscan->iscan, direction, sysscan->slot)) + if (index_getnext_slot(sysscan->iscan, direction, sysscan->slot, NULL)) htup = ExecFetchSlotHeapTuple(sysscan->slot, false, NULL); /* See notes in systable_getnext */ diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c index e493548b68a..f96aeba1b39 100644 --- a/src/backend/access/index/indexam.c +++ b/src/backend/access/index/indexam.c @@ -109,12 +109,14 @@ do { \ static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap, - int prefetch_max); + ParallelIndexScanDesc pscan, bool temp_snap); -static void index_prefetch_tids(IndexScanDesc scan, ScanDirection direction); -static ItemPointer index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_visible); -static void index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible); +static void index_prefetch_tids(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch); +static ItemPointer index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch, bool *all_visible); +static void index_prefetch(IndexScanDesc scan, IndexPrefetch *prefetch, + ItemPointer tid, bool skip_all_visible, bool *all_visible); /* ---------------------------------------------------------------- @@ -223,42 +225,18 @@ index_insert_cleanup(Relation indexRelation, * index_beginscan - start a scan of an index with amgettuple * * Caller must be holding suitable locks on the heap and the index. - * - * prefetch_max determines if prefetching is requested for this index scan, - * and how far ahead we want to prefetch - * - * Setting prefetch_max to 0 disables prefetching for the index scan. We do - * this for two reasons - for scans on system catalogs, and/or for cases where - * prefetching is expected to be pointless (like IOS). - * - * For system catalogs, we usually either scan by a PK value, or we we expect - * only few rows (or rather we don't know how many rows to expect). Also, we - * need to prevent infinite in the get_tablespace_io_concurrency() call - it - * does an index scan internally. So we simply disable prefetching for system - * catalogs. We could deal with this by picking a conservative static target - * (e.g. effective_io_concurrency, capped to something), but places that are - * performance sensitive likely use syscache anyway, and catalogs tend to be - * very small and hot. So we don't bother. - * - * For IOS, we expect to not need most heap pages (that's the whole point of - * IOS, actually), and prefetching them might lead to a lot of wasted I/O. - * - * XXX Not sure the infinite loop can still happen, now that the target lookup - * moved to callers of index_beginscan. */ IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys, - int prefetch_max) + int nkeys, int norderbys) { IndexScanDesc scan; Assert(snapshot != InvalidSnapshot); - scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, - NULL, false, prefetch_max); + scan = index_beginscan_internal(indexRelation, nkeys, norderbys, snapshot, NULL, false); /* * Save additional parameters into the scandesc. Everything else was set @@ -288,8 +266,7 @@ index_beginscan_bitmap(Relation indexRelation, Assert(snapshot != InvalidSnapshot); - /* No prefetch in bitmap scans, prefetch is done by the heap scan. */ - scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false, 0); + scan = index_beginscan_internal(indexRelation, nkeys, 0, snapshot, NULL, false); /* * Save additional parameters into the scandesc. Everything else was set @@ -306,8 +283,7 @@ index_beginscan_bitmap(Relation indexRelation, static IndexScanDesc index_beginscan_internal(Relation indexRelation, int nkeys, int norderbys, Snapshot snapshot, - ParallelIndexScanDesc pscan, bool temp_snap, - int prefetch_max) + ParallelIndexScanDesc pscan, bool temp_snap) { IndexScanDesc scan; @@ -330,31 +306,6 @@ index_beginscan_internal(Relation indexRelation, /* Initialize information for parallel scan. */ scan->parallel_scan = pscan; scan->xs_temp_snap = temp_snap; - scan->indexonly = false; - - /* - * With prefetching requested, initialize the prefetcher state. - * - * FIXME This should really be in the IndexScanState, not IndexScanDesc - * (certainly the queues etc). But index_getnext_tid only gets the scan - * descriptor, so how else would we pass it? Seems like a sign of wrong - * layer doing the prefetching. - */ - if ((prefetch_max > 0) && - (io_direct_flags & IO_DIRECT_DATA) == 0) /* no prefetching for direct I/O */ - { - IndexPrefetch prefetcher = palloc0(sizeof(IndexPrefetchData)); - - prefetcher->queueIndex = 0; - prefetcher->queueStart = 0; - prefetcher->queueEnd = 0; - - prefetcher->prefetchTarget = 0; - prefetcher->prefetchMaxTarget = prefetch_max; - prefetcher->vmBuffer = InvalidBuffer; - - scan->xs_prefetch = prefetcher; - } return scan; } @@ -391,20 +342,6 @@ index_rescan(IndexScanDesc scan, scan->indexRelation->rd_indam->amrescan(scan, keys, nkeys, orderbys, norderbys); - - /* If we're prefetching for this index, maybe reset some of the state. */ - if (scan->xs_prefetch != NULL) - { - IndexPrefetch prefetcher = scan->xs_prefetch; - - prefetcher->queueStart = 0; - prefetcher->queueEnd = 0; - prefetcher->queueIndex = 0; - prefetcher->prefetchDone = false; - - /* restart the incremental ramp-up */ - prefetcher->prefetchTarget = 0; - } } /* ---------------- @@ -433,23 +370,6 @@ index_endscan(IndexScanDesc scan) if (scan->xs_temp_snap) UnregisterSnapshot(scan->xs_snapshot); - /* - * If prefetching was enabled for this scan, log prefetch stats. - * - * FIXME This should really go to EXPLAIN ANALYZE instead. - */ - if (scan->xs_prefetch) - { - IndexPrefetch prefetch = scan->xs_prefetch; - - elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, - prefetch->countAll, - prefetch->countPrefetch, - prefetch->countPrefetch * 100.0 / prefetch->countAll, - prefetch->countSkipCached, - prefetch->countSkipSequential); - } - /* Release the scan data structure itself */ IndexScanEnd(scan); } @@ -595,8 +515,7 @@ index_parallelrescan(IndexScanDesc scan) */ IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, - int norderbys, ParallelIndexScanDesc pscan, - int prefetch_max) + int norderbys, ParallelIndexScanDesc pscan) { Snapshot snapshot; IndexScanDesc scan; @@ -605,7 +524,7 @@ index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, snapshot = RestoreSnapshot(pscan->ps_snapshot_data); RegisterSnapshot(snapshot); scan = index_beginscan_internal(indexrel, nkeys, norderbys, snapshot, - pscan, true, prefetch_max); + pscan, true); /* * Save additional parameters into the scandesc. Everything else was set @@ -727,12 +646,13 @@ index_fetch_heap(IndexScanDesc scan, TupleTableSlot *slot) * ---------------- */ bool -index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot) +index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *slot, + IndexPrefetch *prefetch) { for (;;) { /* Do prefetching (if requested/enabled). */ - index_prefetch_tids(scan, direction); + index_prefetch_tids(scan, direction, prefetch); if (!scan->xs_heap_continue) { @@ -740,7 +660,7 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot * bool all_visible; /* Time to fetch the next TID from the index */ - tid = index_prefetch_get_tid(scan, direction, &all_visible); + tid = index_prefetch_get_tid(scan, direction, prefetch, &all_visible); /* If we're out of index entries, we're done */ if (tid == NULL) @@ -1135,7 +1055,7 @@ index_opclass_options(Relation indrel, AttrNumber attnum, Datum attoptions, * does not require a sequential pattern). */ static bool -index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) +index_prefetch_is_sequential(IndexPrefetch *prefetch, BlockNumber block) { int idx; @@ -1270,9 +1190,9 @@ index_prefetch_is_sequential(IndexPrefetch prefetch, BlockNumber block) * what index_prefetch_is_sequential does. */ static bool -index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) +index_prefetch_add_cache(IndexPrefetch *prefetch, BlockNumber block) { - PrefetchCacheEntry *entry; + IndexPrefetchCacheEntry *entry; /* map the block number the the LRU */ int lru = hash_uint32(block) % PREFETCH_LRU_COUNT; @@ -1419,9 +1339,9 @@ index_prefetch_add_cache(IndexPrefetch prefetch, BlockNumber block) * value once in a while, and see what happens. */ static void -index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool *all_visible) +index_prefetch(IndexScanDesc scan, IndexPrefetch *prefetch, + ItemPointer tid, bool skip_all_visible, bool *all_visible) { - IndexPrefetch prefetch = scan->xs_prefetch; BlockNumber block; /* by default not all visible (or we didn't check) */ @@ -1502,33 +1422,33 @@ index_prefetch(IndexScanDesc scan, ItemPointer tid, bool skip_all_visible, bool * ---------------- */ ItemPointer -index_getnext_tid(IndexScanDesc scan, ScanDirection direction) +index_getnext_tid(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch) { bool all_visible; /* ignored */ /* Do prefetching (if requested/enabled). */ - index_prefetch_tids(scan, direction); + index_prefetch_tids(scan, direction, prefetch); /* Read the TID from the queue (or directly from the index). */ - return index_prefetch_get_tid(scan, direction, &all_visible); + return index_prefetch_get_tid(scan, direction, prefetch, &all_visible); } ItemPointer -index_getnext_tid_vm(IndexScanDesc scan, ScanDirection direction, bool *all_visible) +index_getnext_tid_vm(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch, bool *all_visible) { /* Do prefetching (if requested/enabled). */ - index_prefetch_tids(scan, direction); + index_prefetch_tids(scan, direction, prefetch); /* Read the TID from the queue (or directly from the index). */ - return index_prefetch_get_tid(scan, direction, all_visible); + return index_prefetch_get_tid(scan, direction, prefetch, all_visible); } static void -index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) +index_prefetch_tids(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch) { - /* for convenience */ - IndexPrefetch prefetch = scan->xs_prefetch; - /* * If the prefetching is still active (i.e. enabled and we still * haven't finished reading TIDs from the scan), read enough TIDs into @@ -1577,7 +1497,7 @@ index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) * XXX index_getnext_tid_prefetch is only called for IOS (for now), * so skip prefetching of all-visible pages. */ - index_prefetch(scan, tid, scan->indexonly, &all_visible); + index_prefetch(scan, prefetch, tid, prefetch->indexonly, &all_visible); prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)].tid = *tid; prefetch->queueItems[PREFETCH_QUEUE_INDEX(prefetch->queueEnd)].all_visible = all_visible; @@ -1587,11 +1507,9 @@ index_prefetch_tids(IndexScanDesc scan, ScanDirection direction) } static ItemPointer -index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, bool *all_visible) +index_prefetch_get_tid(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch, bool *all_visible) { - /* for convenience */ - IndexPrefetch prefetch = scan->xs_prefetch; - /* * With prefetching enabled (even if we already finished reading * all TIDs from the index scan), we need to return a TID from the diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 15fa3211667..0a136db6712 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -770,18 +770,18 @@ check_exclusion_or_unique_constraint(Relation heap, Relation index, /* * May have to restart scan from this point if a potential conflict is * found. - * - * XXX Should this do index prefetch? Probably not worth it for unique - * constraints, I guess? Otherwise we should calculate prefetch_target - * just like in nodeIndexscan etc. */ retry: conflict = false; found_self = false; - index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0, 0); + index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0); index_rescan(index_scan, scankeys, indnkeyatts, NULL, 0); - while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot)) + /* + * XXX Would be nice to also benefit from prefetching here. All we need to + * do is instantiate the prefetcher, I guess. + */ + while (index_getnext_slot(index_scan, ForwardScanDirection, existing_slot, NULL)) { TransactionId xwait; XLTW_Oper reason_wait; diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 91676ccff95..9498b00fa64 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -204,21 +204,21 @@ RelationFindReplTupleByIndex(Relation rel, Oid idxoid, /* Build scan key. */ skey_attoff = build_replindex_scan_key(skey, rel, idxrel, searchslot); - /* Start an index scan. - * - * XXX Should this do index prefetching? We're looking for a single tuple, - * probably using a PK / UNIQUE index, so does not seem worth it. If we - * reconsider this, calclate prefetch_target like in nodeIndexscan. - */ - scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0, 0); + /* Start an index scan. */ + scan = index_beginscan(rel, idxrel, &snap, skey_attoff, 0); retry: found = false; index_rescan(scan, skey, skey_attoff, NULL, 0); - /* Try to find the tuple */ - while (index_getnext_slot(scan, ForwardScanDirection, outslot)) + /* + * Try to find the tuple + * + * XXX Would be nice to also benefit from prefetching here. All we need to + * do is instantiate the prefetcher, I guess. + */ + while (index_getnext_slot(scan, ForwardScanDirection, outslot, NULL)) { /* * Avoid expensive equality check if the index is primary key or diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index b6660c10a63..a7eadaf3db2 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -65,8 +65,8 @@ IndexOnlyNext(IndexOnlyScanState *node) IndexScanDesc scandesc; TupleTableSlot *slot; ItemPointer tid; - Relation heapRel = node->ss.ss_currentRelation; - bool all_visible; + IndexPrefetch *prefetch; + bool all_visible; /* * extract necessary information from index scan node @@ -80,52 +80,22 @@ IndexOnlyNext(IndexOnlyScanState *node) direction = ScanDirectionCombine(estate->es_direction, ((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir); scandesc = node->ioss_ScanDesc; + prefetch = node->ioss_prefetch; econtext = node->ss.ps.ps_ExprContext; slot = node->ss.ss_ScanTupleSlot; if (scandesc == NULL) { - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - * - * XXX Maybe reduce the value with parallel workers? - */ - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); - /* * We reach here if the index only scan is not parallel, or if we're * serially executing an index only scan that was planned to be * parallel. - * - * XXX Maybe we should enable prefetching, but prefetch only pages that - * are not all-visible (but checking that from the index code seems like - * a violation of layering etc). - * - * XXX This might lead to IOS being slower than plain index scan, if the - * table has a lot of pages that need recheck. */ scandesc = index_beginscan(node->ss.ss_currentRelation, node->ioss_RelationDesc, estate->es_snapshot, node->ioss_NumScanKeys, - node->ioss_NumOrderByKeys, - prefetch_max); - - /* - * Remember this is index-only scan, because of prefetching. Not the most - * elegant way to pass this info. - */ - scandesc->indexonly = true; + node->ioss_NumOrderByKeys); node->ioss_ScanDesc = scandesc; @@ -149,7 +119,7 @@ IndexOnlyNext(IndexOnlyScanState *node) /* * OK, now that we have what we need, fetch the next tuple. */ - while ((tid = index_getnext_tid_vm(scandesc, direction, &all_visible)) != NULL) + while ((tid = index_getnext_tid_vm(scandesc, direction, prefetch, &all_visible)) != NULL) { bool tuple_from_heap = false; @@ -389,6 +359,16 @@ ExecReScanIndexOnlyScan(IndexOnlyScanState *node) node->ioss_ScanKeys, node->ioss_NumScanKeys, node->ioss_OrderByKeys, node->ioss_NumOrderByKeys); + /* also reset the prefetcher, so that we start from scratch */ + if (node->ioss_prefetch) + { + IndexPrefetch *prefetch = node->ioss_prefetch; + + prefetch->queueIndex = 0; + prefetch->queueStart = 0; + prefetch->queueEnd = 0; + } + ExecScanReScan(&node->ss); } @@ -417,14 +397,22 @@ ExecEndIndexOnlyScan(IndexOnlyScanState *node) } /* Release VM buffer pin from prefetcher, if any. */ - if (indexScanDesc && indexScanDesc->xs_prefetch) + if (node->ioss_prefetch) { - IndexPrefetch indexPrefetch = indexScanDesc->xs_prefetch; + IndexPrefetch *prefetch = node->ioss_prefetch; + + /* XXX some debug info */ + elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, + prefetch->countAll, + prefetch->countPrefetch, + prefetch->countPrefetch * 100.0 / prefetch->countAll, + prefetch->countSkipCached, + prefetch->countSkipSequential); - if (indexPrefetch->vmBuffer != InvalidBuffer) + if (prefetch->vmBuffer != InvalidBuffer) { - ReleaseBuffer(indexPrefetch->vmBuffer); - indexPrefetch->vmBuffer = InvalidBuffer; + ReleaseBuffer(prefetch->vmBuffer); + prefetch->vmBuffer = InvalidBuffer; } } @@ -652,6 +640,63 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags) indexstate->ioss_RuntimeContext = NULL; } + /* + * Also initialize index prefetcher. + * + * XXX No prefetching for direct I/O. + */ + if ((io_direct_flags & IO_DIRECT_DATA) == 0) + { + int prefetch_max; + Relation heapRel = indexstate->ss.ss_currentRelation; + + /* + * Determine number of heap pages to prefetch for this index. This is + * essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + * + * XXX Maybe reduce the value with parallel workers? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + indexstate->ss.ps.plan->plan_rows); + + /* + * We reach here if the index only scan is not parallel, or if we're + * serially executing an index only scan that was planned to be + * parallel. + * + * XXX Maybe we should enable prefetching, but prefetch only pages that + * are not all-visible (but checking that from the index code seems like + * a violation of layering etc). + * + * XXX This might lead to IOS being slower than plain index scan, if the + * table has a lot of pages that need recheck. + * + * Remember this is index-only scan, because of prefetching. Not the most + * elegant way to pass this info. + */ + if (prefetch_max > 0) + { + IndexPrefetch *prefetch = palloc0(sizeof(IndexPrefetch)); + + prefetch->queueIndex = 0; + prefetch->queueStart = 0; + prefetch->queueEnd = 0; + + prefetch->prefetchTarget = 0; + prefetch->prefetchMaxTarget = prefetch_max; + prefetch->vmBuffer = InvalidBuffer; + prefetch->indexonly = true; + + indexstate->ioss_prefetch = prefetch; + } + } + /* * all done. */ @@ -694,24 +739,6 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; - Relation heapRel = node->ss.ss_currentRelation; - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - * - * XXX Maybe reduce the value with parallel workers? - */ - - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->ioss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -724,8 +751,7 @@ ExecIndexOnlyScanInitializeDSM(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan, - prefetch_max); + piscan); node->ioss_ScanDesc->xs_want_itup = true; node->ioss_VMBuffer = InvalidBuffer; @@ -763,23 +789,6 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; - Relation heapRel = node->ss.ss_currentRelation; - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - * - * XXX Maybe reduce the value with parallel workers? - */ - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->ioss_ScanDesc = @@ -787,8 +796,7 @@ ExecIndexOnlyScanInitializeWorker(IndexOnlyScanState *node, node->ioss_RelationDesc, node->ioss_NumScanKeys, node->ioss_NumOrderByKeys, - piscan, - prefetch_max); + piscan); node->ioss_ScanDesc->xs_want_itup = true; /* diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index a5f5394ef49..b3282ec5a75 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -86,7 +86,7 @@ IndexNext(IndexScanState *node) ScanDirection direction; IndexScanDesc scandesc; TupleTableSlot *slot; - Relation heapRel = node->ss.ss_currentRelation; + IndexPrefetch *prefetch; /* * extract necessary information from index scan node @@ -100,26 +100,12 @@ IndexNext(IndexScanState *node) direction = ScanDirectionCombine(estate->es_direction, ((IndexScan *) node->ss.ps.plan)->indexorderdir); scandesc = node->iss_ScanDesc; + prefetch = node->iss_prefetch; econtext = node->ss.ps.ps_ExprContext; slot = node->ss.ss_ScanTupleSlot; if (scandesc == NULL) { - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index scan. This - * is essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - */ - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); - /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -128,8 +114,7 @@ IndexNext(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys, - prefetch_max); + node->iss_NumOrderByKeys); node->iss_ScanDesc = scandesc; @@ -146,7 +131,7 @@ IndexNext(IndexScanState *node) /* * ok, now that we have what we need, fetch the next tuple. */ - while (index_getnext_slot(scandesc, direction, slot)) + while (index_getnext_slot(scandesc, direction, slot, prefetch)) { CHECK_FOR_INTERRUPTS(); @@ -195,7 +180,7 @@ IndexNextWithReorder(IndexScanState *node) Datum *lastfetched_vals; bool *lastfetched_nulls; int cmp; - Relation heapRel = node->ss.ss_currentRelation; + IndexPrefetch *prefetch; estate = node->ss.ps.state; @@ -212,26 +197,12 @@ IndexNextWithReorder(IndexScanState *node) Assert(ScanDirectionIsForward(estate->es_direction)); scandesc = node->iss_ScanDesc; + prefetch = node->iss_prefetch; econtext = node->ss.ps.ps_ExprContext; slot = node->ss.ss_ScanTupleSlot; if (scandesc == NULL) { - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - */ - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); - /* * We reach here if the index scan is not parallel, or if we're * serially executing an index scan that was planned to be parallel. @@ -240,8 +211,7 @@ IndexNextWithReorder(IndexScanState *node) node->iss_RelationDesc, estate->es_snapshot, node->iss_NumScanKeys, - node->iss_NumOrderByKeys, - prefetch_max); + node->iss_NumOrderByKeys); node->iss_ScanDesc = scandesc; @@ -294,7 +264,7 @@ IndexNextWithReorder(IndexScanState *node) * Fetch next tuple from the index. */ next_indextuple: - if (!index_getnext_slot(scandesc, ForwardScanDirection, slot)) + if (!index_getnext_slot(scandesc, ForwardScanDirection, slot, prefetch)) { /* * No more tuples from the index. But we still need to drain any @@ -623,6 +593,16 @@ ExecReScanIndexScan(IndexScanState *node) node->iss_OrderByKeys, node->iss_NumOrderByKeys); node->iss_ReachedEnd = false; + /* also reset the prefetcher, so that we start from scratch */ + if (node->iss_prefetch) + { + IndexPrefetch *prefetch = node->iss_prefetch; + + prefetch->queueIndex = 0; + prefetch->queueStart = 0; + prefetch->queueEnd = 0; + } + ExecScanReScan(&node->ss); } @@ -829,6 +809,19 @@ ExecEndIndexScan(IndexScanState *node) indexRelationDesc = node->iss_RelationDesc; indexScanDesc = node->iss_ScanDesc; + /* XXX nothing to free, but print some debug info */ + if (node->iss_prefetch) + { + IndexPrefetch *prefetch = node->iss_prefetch; + + elog(LOG, "index prefetch stats: requests " UINT64_FORMAT " prefetches " UINT64_FORMAT " (%f) skip cached " UINT64_FORMAT " sequential " UINT64_FORMAT, + prefetch->countAll, + prefetch->countPrefetch, + prefetch->countPrefetch * 100.0 / prefetch->countAll, + prefetch->countSkipCached, + prefetch->countSkipSequential); + } + /* * close the index relation (no-op if we didn't open it) */ @@ -1101,6 +1094,45 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags) indexstate->iss_RuntimeContext = NULL; } + /* + * Also initialize index prefetcher. + * + * XXX No prefetching for direct I/O. + */ + if ((io_direct_flags & IO_DIRECT_DATA) == 0) + { + int prefetch_max; + Relation heapRel = indexstate->ss.ss_currentRelation; + + /* + * Determine number of heap pages to prefetch for this index scan. This + * is essentially just effective_io_concurrency for the table (or the + * tablespace it's in). + * + * XXX Should this also look at plan.plan_rows and maybe cap the target + * to that? Pointless to prefetch more than we expect to use. Or maybe + * just reset to that value during prefetching, after reading the next + * index page (or rather after rescan)? + */ + prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), + indexstate->ss.ps.plan->plan_rows); + + if (prefetch_max > 0) + { + IndexPrefetch *prefetch = palloc0(sizeof(IndexPrefetch)); + + prefetch->queueIndex = 0; + prefetch->queueStart = 0; + prefetch->queueEnd = 0; + + prefetch->prefetchTarget = 0; + prefetch->prefetchMaxTarget = prefetch_max; + prefetch->vmBuffer = InvalidBuffer; + + indexstate->iss_prefetch = prefetch; + } + } + /* * all done. */ @@ -1697,24 +1729,6 @@ ExecIndexScanInitializeDSM(IndexScanState *node, { EState *estate = node->ss.ps.state; ParallelIndexScanDesc piscan; - Relation heapRel = node->ss.ss_currentRelation; - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - * - * XXX Maybe reduce the value with parallel workers? - */ - - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); piscan = shm_toc_allocate(pcxt->toc, node->iss_PscanLen); index_parallelscan_initialize(node->ss.ss_currentRelation, @@ -1727,8 +1741,7 @@ ExecIndexScanInitializeDSM(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan, - prefetch_max); + piscan); /* * If no run-time keys to calculate or they are ready, go ahead and pass @@ -1764,23 +1777,6 @@ ExecIndexScanInitializeWorker(IndexScanState *node, ParallelWorkerContext *pwcxt) { ParallelIndexScanDesc piscan; - Relation heapRel = node->ss.ss_currentRelation; - int prefetch_max; - - /* - * Determine number of heap pages to prefetch for this index. This is - * essentially just effective_io_concurrency for the table (or the - * tablespace it's in). - * - * XXX Should this also look at plan.plan_rows and maybe cap the target - * to that? Pointless to prefetch more than we expect to use. Or maybe - * just reset to that value during prefetching, after reading the next - * index page (or rather after rescan)? - * - * XXX Maybe reduce the value with parallel workers? - */ - prefetch_max = Min(get_tablespace_io_concurrency(heapRel->rd_rel->reltablespace), - node->ss.ps.plan->plan_rows); piscan = shm_toc_lookup(pwcxt->toc, node->ss.ps.plan->plan_node_id, false); node->iss_ScanDesc = @@ -1788,8 +1784,7 @@ ExecIndexScanInitializeWorker(IndexScanState *node, node->iss_RelationDesc, node->iss_NumScanKeys, node->iss_NumOrderByKeys, - piscan, - prefetch_max); + piscan); /* * If no run-time keys to calculate or they are ready, go ahead and pass diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 8b662d371dd..b5c79359425 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -6289,16 +6289,15 @@ get_actual_variable_endpoint(Relation heapRel, InitNonVacuumableSnapshot(SnapshotNonVacuumable, GlobalVisTestFor(heapRel)); - /* XXX Maybe should do prefetching using the default prefetch parameters? */ index_scan = index_beginscan(heapRel, indexRel, &SnapshotNonVacuumable, - 1, 0, 0); + 1, 0); /* Set it up for index-only scan */ index_scan->xs_want_itup = true; index_rescan(index_scan, scankeys, 1, NULL, 0); /* Fetch first/next tuple in specified direction */ - while ((tid = index_getnext_tid(index_scan, indexscandir)) != NULL) + while ((tid = index_getnext_tid(index_scan, indexscandir, NULL)) != NULL) { BlockNumber block = ItemPointerGetBlockNumber(tid); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index f6882f644d2..c0c46d7a05f 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -129,6 +129,110 @@ typedef struct IndexOrderByDistance bool isnull; } IndexOrderByDistance; + + +/* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. Doesn't need to be perfectly accurate, but we + * aim to make false positives/negatives reasonably low. + */ +typedef struct IndexPrefetchCacheEntry { + BlockNumber block; + uint64 request; +} IndexPrefetchCacheEntry; + +/* + * Size of the cache of recently prefetched blocks - shouldn't be too + * small or too large. 1024 seems about right, it covers ~8MB of data. + * It's somewhat arbitrary, there's no particular formula saying it + * should not be higher/lower. + * + * The cache is structured as an array of small LRU caches, so the total + * size needs to be a multiple of LRU size. The LRU should be tiny to + * keep linear search cheap enough. + * + * XXX Maybe we could consider effective_cache_size or something? + */ +#define PREFETCH_LRU_SIZE 8 +#define PREFETCH_LRU_COUNT 128 +#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) + +/* + * Used to detect sequential patterns (and disable prefetching). + */ +#define PREFETCH_QUEUE_HISTORY 8 +#define PREFETCH_SEQ_PATTERN_BLOCKS 4 + +typedef struct IndexPrefetchEntry +{ + ItemPointerData tid; + bool all_visible; +} IndexPrefetchEntry; + +typedef struct IndexPrefetch +{ + /* + * XXX We need to disable this in some cases (e.g. when using index-only + * scans, we don't want to prefetch pages). Or maybe we should prefetch + * only pages that are not all-visible, that'd be even better. + */ + int prefetchTarget; /* how far we should be prefetching */ + int prefetchMaxTarget; /* maximum prefetching distance */ + int prefetchReset; /* reset to this distance on rescan */ + bool prefetchDone; /* did we get all TIDs from the index? */ + + /* runtime statistics */ + uint64 countAll; /* all prefetch requests */ + uint64 countPrefetch; /* actual prefetches */ + uint64 countSkipSequential; + uint64 countSkipCached; + + /* used when prefetching index-only scans */ + bool indexonly; + Buffer vmBuffer; + + /* + * Queue of TIDs to prefetch. + * + * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler + * than dynamically adjusting for custom values. + */ + IndexPrefetchEntry queueItems[MAX_IO_CONCURRENCY]; + uint64 queueIndex; /* next TID to prefetch */ + uint64 queueStart; /* first valid TID in queue */ + uint64 queueEnd; /* first invalid (empty) TID in queue */ + + /* + * A couple of last prefetched blocks, used to check for certain access + * pattern and skip prefetching - e.g. for sequential access). + * + * XXX Separate from the main queue, because we only want to compare the + * block numbers, not the whole TID. In sequential access it's likely we + * read many items from each page, and we don't want to check many items + * (as that is much more expensive). + */ + BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; + uint64 blockIndex; /* index in the block (points to the first + * empty entry)*/ + + /* + * Cache of recently prefetched blocks, organized as a hash table of + * small LRU caches. + */ + uint64 prefetchReqNumber; + IndexPrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; + +} IndexPrefetch; + +#define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) +#define PREFETCH_QUEUE_EMPTY(p) ((p)->queueEnd == (p)->queueIndex) +#define PREFETCH_ENABLED(p) ((p) && ((p)->prefetchMaxTarget > 0)) +#define PREFETCH_FULL(p) ((p)->queueEnd - (p)->queueIndex == (p)->prefetchTarget) +#define PREFETCH_DONE(p) ((p) && ((p)->prefetchDone && PREFETCH_QUEUE_EMPTY(p))) +#define PREFETCH_ACTIVE(p) (PREFETCH_ENABLED(p) && !(p)->prefetchDone) +#define PREFETCH_BLOCK_INDEX(v) ((v) % PREFETCH_QUEUE_HISTORY) + + /* * generalized index_ interface routines (in indexam.c) */ @@ -155,8 +259,7 @@ extern void index_insert_cleanup(Relation indexRelation, extern IndexScanDesc index_beginscan(Relation heapRelation, Relation indexRelation, Snapshot snapshot, - int nkeys, int norderbys, - int prefetch_max); + int nkeys, int norderbys); extern IndexScanDesc index_beginscan_bitmap(Relation indexRelation, Snapshot snapshot, int nkeys); @@ -173,17 +276,19 @@ extern void index_parallelscan_initialize(Relation heapRelation, extern void index_parallelrescan(IndexScanDesc scan); extern IndexScanDesc index_beginscan_parallel(Relation heaprel, Relation indexrel, int nkeys, int norderbys, - ParallelIndexScanDesc pscan, - int prefetch_max); + ParallelIndexScanDesc pscan); extern ItemPointer index_getnext_tid(IndexScanDesc scan, - ScanDirection direction); + ScanDirection direction, + IndexPrefetch *prefetch); extern ItemPointer index_getnext_tid_vm(IndexScanDesc scan, ScanDirection direction, + IndexPrefetch *prefetch, bool *all_visible); struct TupleTableSlot; extern bool index_fetch_heap(IndexScanDesc scan, struct TupleTableSlot *slot); extern bool index_getnext_slot(IndexScanDesc scan, ScanDirection direction, - struct TupleTableSlot *slot); + struct TupleTableSlot *slot, + IndexPrefetch *prefetch); extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap); extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info, @@ -238,104 +343,4 @@ extern HeapTuple systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction); extern void systable_endscan_ordered(SysScanDesc sysscan); -/* - * Cache of recently prefetched blocks, organized as a hash table of - * small LRU caches. Doesn't need to be perfectly accurate, but we - * aim to make false positives/negatives reasonably low. - */ -typedef struct PrefetchCacheEntry { - BlockNumber block; - uint64 request; -} PrefetchCacheEntry; - -/* - * Size of the cache of recently prefetched blocks - shouldn't be too - * small or too large. 1024 seems about right, it covers ~8MB of data. - * It's somewhat arbitrary, there's no particular formula saying it - * should not be higher/lower. - * - * The cache is structured as an array of small LRU caches, so the total - * size needs to be a multiple of LRU size. The LRU should be tiny to - * keep linear search cheap enough. - * - * XXX Maybe we could consider effective_cache_size or something? - */ -#define PREFETCH_LRU_SIZE 8 -#define PREFETCH_LRU_COUNT 128 -#define PREFETCH_CACHE_SIZE (PREFETCH_LRU_SIZE * PREFETCH_LRU_COUNT) - -/* - * Used to detect sequential patterns (and disable prefetching). - */ -#define PREFETCH_QUEUE_HISTORY 8 -#define PREFETCH_SEQ_PATTERN_BLOCKS 4 - -typedef struct PrefetchEntry -{ - ItemPointerData tid; - bool all_visible; -} PrefetchEntry; - -typedef struct IndexPrefetchData -{ - /* - * XXX We need to disable this in some cases (e.g. when using index-only - * scans, we don't want to prefetch pages). Or maybe we should prefetch - * only pages that are not all-visible, that'd be even better. - */ - int prefetchTarget; /* how far we should be prefetching */ - int prefetchMaxTarget; /* maximum prefetching distance */ - int prefetchReset; /* reset to this distance on rescan */ - bool prefetchDone; /* did we get all TIDs from the index? */ - - /* runtime statistics */ - uint64 countAll; /* all prefetch requests */ - uint64 countPrefetch; /* actual prefetches */ - uint64 countSkipSequential; - uint64 countSkipCached; - - /* used when prefetching index-only scans */ - Buffer vmBuffer; - - /* - * Queue of TIDs to prefetch. - * - * XXX Sizing for MAX_IO_CONCURRENCY may be overkill, but it seems simpler - * than dynamically adjusting for custom values. - */ - PrefetchEntry queueItems[MAX_IO_CONCURRENCY]; - uint64 queueIndex; /* next TID to prefetch */ - uint64 queueStart; /* first valid TID in queue */ - uint64 queueEnd; /* first invalid (empty) TID in queue */ - - /* - * A couple of last prefetched blocks, used to check for certain access - * pattern and skip prefetching - e.g. for sequential access). - * - * XXX Separate from the main queue, because we only want to compare the - * block numbers, not the whole TID. In sequential access it's likely we - * read many items from each page, and we don't want to check many items - * (as that is much more expensive). - */ - BlockNumber blockItems[PREFETCH_QUEUE_HISTORY]; - uint64 blockIndex; /* index in the block (points to the first - * empty entry)*/ - - /* - * Cache of recently prefetched blocks, organized as a hash table of - * small LRU caches. - */ - uint64 prefetchReqNumber; - PrefetchCacheEntry prefetchCache[PREFETCH_CACHE_SIZE]; - -} IndexPrefetchData; - -#define PREFETCH_QUEUE_INDEX(a) ((a) % (MAX_IO_CONCURRENCY)) -#define PREFETCH_QUEUE_EMPTY(p) ((p)->queueEnd == (p)->queueIndex) -#define PREFETCH_ENABLED(p) ((p) && ((p)->prefetchMaxTarget > 0)) -#define PREFETCH_FULL(p) ((p)->queueEnd - (p)->queueIndex == (p)->prefetchTarget) -#define PREFETCH_DONE(p) ((p) && ((p)->prefetchDone && PREFETCH_QUEUE_EMPTY(p))) -#define PREFETCH_ACTIVE(p) (PREFETCH_ENABLED(p) && !(p)->prefetchDone) -#define PREFETCH_BLOCK_INDEX(v) ((v) % PREFETCH_QUEUE_HISTORY) - #endif /* GENAM_H */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index d5903492c6e..d03360eac04 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -106,12 +106,6 @@ typedef struct IndexFetchTableData Relation rel; } IndexFetchTableData; -/* - * Forward declarations, defined in genam.h. - */ -typedef struct IndexPrefetchData IndexPrefetchData; -typedef struct IndexPrefetchData *IndexPrefetch; - /* * We use the same IndexScanDescData structure for both amgettuple-based * and amgetbitmap-based index scans. Some fields are only relevant in @@ -135,7 +129,6 @@ typedef struct IndexScanDescData bool ignore_killed_tuples; /* do not return killed entries */ bool xactStartedInRecovery; /* prevents killing/seeing killed * tuples */ - bool indexonly; /* is this index-only scan? */ /* index access method's private state */ void *opaque; /* access-method-specific info */ @@ -169,9 +162,6 @@ typedef struct IndexScanDescData bool *xs_orderbynulls; bool xs_recheckorderby; - /* prefetching state (or NULL if disabled for this scan) */ - IndexPrefetchData *xs_prefetch; - /* parallel index scan information, in shared memory */ struct ParallelIndexScanDescData *parallel_scan; } IndexScanDescData; diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 5d7f17dee07..8745453a5b4 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1529,6 +1529,7 @@ typedef struct bool *elem_nulls; /* array of num_elems is-null flags */ } IndexArrayKeyInfo; + /* ---------------- * IndexScanState information * @@ -1580,6 +1581,8 @@ typedef struct IndexScanState bool *iss_OrderByTypByVals; int16 *iss_OrderByTypLens; Size iss_PscanLen; + + IndexPrefetch *iss_prefetch; } IndexScanState; /* ---------------- @@ -1618,6 +1621,7 @@ typedef struct IndexOnlyScanState TupleTableSlot *ioss_TableSlot; Buffer ioss_VMBuffer; Size ioss_PscanLen; + IndexPrefetch *ioss_prefetch; } IndexOnlyScanState; /* ---------------- -- 2.41.0 ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-12-18 21:00 Robert Haas <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Robert Haas @ 2023-12-18 21:00 UTC (permalink / raw) To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> On Sat, Dec 9, 2023 at 1:08 PM Tomas Vondra <[email protected]> wrote: > But there's a layering problem that I don't know how to solve - I don't > see how we could make indexam.c entirely oblivious to the prefetching, > and move it entirely to the executor. Because how else would you know > what to prefetch? Yeah, that seems impossible. Some thoughts: * I think perhaps the subject line of this thread is misleading. It doesn't seem like there is any index prefetching going on here at all, and there couldn't be, unless you extended the index AM API with new methods. What you're actually doing is prefetching heap pages that will be needed by a scan of the index. I think this confusing naming has propagated itself into some parts of the patch, e.g. index_prefetch() reads *from the heap* which is not at all clear from the comment saying "Prefetch the TID, unless it's sequential or recently prefetched." You're not prefetching the TID: you're prefetching the heap tuple to which the TID points. That's not an academic distinction IMHO -- the TID would be stored in the index, so if we were prefetching the TID, we'd have to be reading index pages, not heap pages. * Regarding layering, my first thought was that the changes to index_getnext_tid() and index_getnext_slot() are sensible: read ahead by some number of TIDs, keep the TIDs you've fetched in an array someplace, use that to drive prefetching of blocks on disk, and return the previously-read TIDs from the queue without letting the caller know that the queue exists. I think that's the obvious design for a feature of this type, to the point where I don't really see that there's a viable alternative design. Driving something down into the individual index AMs would make sense if you wanted to prefetch *from the indexes*, but it's unnecessary otherwise, and best avoided. * But that said, the skip_all_visible flag passed down to index_prefetch() looks like a VERY strong sign that the layering here is not what it should be. Right now, when some code calls index_getnext_tid(), that function does not need to know or care whether the caller is going to fetch the heap tuple or not. But with this patch, the code does need to care. So knowledge of the executor concept of an index-only scan trickles down into indexam.c, which now has to be able to make decisions that are consistent with the ones that the executor will make. That doesn't seem good at all. * I think it might make sense to have two different prefetching schemes. Ideally they could share some structure. If a caller is using index_getnext_slot(), then it's easy for prefetching to be fully transparent. The caller can just ask for TIDs and the prefetching distance and TID queue can be fully under the control of something that is hidden from the caller. But when using index_getnext_tid(), the caller needs to have an opportunity to evaluate each TID and decide whether we even want the heap tuple. If yes, then we feed that TID to the prefetcher; if no, we don't. That way, we're not replicating executor logic in lower-level code. However, that also means that the IOS logic needs to be aware that this TID queue exists and interact with whatever controls the prefetch distance. Perhaps after calling index_getnext_tid() you call index_prefetcher_put_tid(prefetcher, tid, bool fetch_heap_tuple) and then you call index_prefetcher_get_tid() to drain the queue. Perhaps also the prefetcher has a "fill" callback that gets invoked when the TID queue isn't as full as the prefetcher wants it to be. Then index_getnext_slot() can just install a trivial fill callback that says index_prefetecher_put_tid(prefetcher, index_getnext_tid(...), true), but IOS can use a more sophisticated callback that checks the VM to determine what to pass for the third argument. * I realize that I'm being a little inconsistent in what I just said, because in the first bullet point I said that this wasn't really index prefetching, and now I'm proposing function names that still start with index_prefetch. It's not entirely clear to me what the best thing to do about the terminology is here -- could it be a heap prefetcher, or a TID prefetcher, or an index scan prefetcher? I don't really know, but whatever we can do to make the naming more clear seems like a really good idea. Maybe there should be a clearer separation between the queue of TIDs that we're going to return from the index and the queue of blocks that we want to prefetch to get the corresponding heap tuples -- making that separation crisper might ease some of the naming issues. * Not that I want to be critical because I think this is a great start on an important project, but it does look like there's an awful lot of stuff here that still needs to be sorted out before it would be reasonable to think of committing this, both in terms of design decisions and just general polish. There's a lot of stuff marked with XXX and I think that's great because most of those seem to be good questions but that does leave the, err, small problem of figuring out the answers. index_prefetch_is_sequential() makes me really nervous because it seems to depend an awful lot on whether the OS is doing prefetching, and how the OS is doing prefetching, and I think those might not be consistent across all systems and kernel versions. Similarly with index_prefetch(). There's a lot of "magical" assumptions here. Even index_prefetch_add_cache() has this problem -- the function assumes that it's OK if we sometimes fail to detect a duplicate prefetch request, which makes sense, but under what circumstances is it necessary to detect duplicates and in what cases is it optional? The function comments are silent about that, which makes it hard to assess whether the algorithm is good enough. * In terms of polish, one thing I noticed is that index_getnext_slot() calls index_prefetch_tids() even when scan->xs_heap_continue is set, which seems like it must be a waste, since we can't really need to kick off more prefetch requests halfway through a HOT chain referenced by a single index tuple, can we? Also, blks_prefetch_rounds doesn't seem to be used anywhere, and neither that nor blks_prefetches are documented. In fact there's no new documentation at all, which seems probably not right. That's partly because there are no new GUCs, which I feel like typically for a feature like this would be the place where the feature behavior would be mentioned in the documentation. I don't think it's a good idea to tie the behavior of this feature to effective_io_concurrency partly because it's usually a bad idea to make one setting control multiple different things, but perhaps even more because effective_io_concurrency doesn't actually work in a useful way AFAICT and people typically have to set it to some very artificially large value compared to how much real I/O parallelism they have. So probably there should be new GUCs with hopefully-better semantics, but at least the documentation for any existing ones would need updating, I would think. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-12-20 01:41 Tomas Vondra <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Tomas Vondra @ 2023-12-20 01:41 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> On 12/18/23 22:00, Robert Haas wrote: > On Sat, Dec 9, 2023 at 1:08 PM Tomas Vondra > <[email protected]> wrote: >> But there's a layering problem that I don't know how to solve - I don't >> see how we could make indexam.c entirely oblivious to the prefetching, >> and move it entirely to the executor. Because how else would you know >> what to prefetch? > > Yeah, that seems impossible. > > Some thoughts: > > * I think perhaps the subject line of this thread is misleading. It > doesn't seem like there is any index prefetching going on here at all, > and there couldn't be, unless you extended the index AM API with new > methods. What you're actually doing is prefetching heap pages that > will be needed by a scan of the index. I think this confusing naming > has propagated itself into some parts of the patch, e.g. > index_prefetch() reads *from the heap* which is not at all clear from > the comment saying "Prefetch the TID, unless it's sequential or > recently prefetched." You're not prefetching the TID: you're > prefetching the heap tuple to which the TID points. That's not an > academic distinction IMHO -- the TID would be stored in the index, so > if we were prefetching the TID, we'd have to be reading index pages, > not heap pages. Yes, that's a fair complaint. I think the naming is mostly obsolete - the prefetching initially happened way way lower - in the index AMs. It was prefetching the heap pages, ofc, but it kinda seemed reasonable to call it "index prefetching". And even now it's called from indexam.c where most functions start with "index_". But I'll think about some better / cleared name. > > * Regarding layering, my first thought was that the changes to > index_getnext_tid() and index_getnext_slot() are sensible: read ahead > by some number of TIDs, keep the TIDs you've fetched in an array > someplace, use that to drive prefetching of blocks on disk, and return > the previously-read TIDs from the queue without letting the caller > know that the queue exists. I think that's the obvious design for a > feature of this type, to the point where I don't really see that > there's a viable alternative design. I agree. > Driving something down into the individual index AMs would make sense > if you wanted to prefetch *from the indexes*, but it's unnecessary > otherwise, and best avoided. > Right. In fact, the patch moved exactly in the opposite direction - it was originally done at the AM level, and moved up. First to indexam.c, then even more to the executor. > * But that said, the skip_all_visible flag passed down to > index_prefetch() looks like a VERY strong sign that the layering here > is not what it should be. Right now, when some code calls > index_getnext_tid(), that function does not need to know or care > whether the caller is going to fetch the heap tuple or not. But with > this patch, the code does need to care. So knowledge of the executor > concept of an index-only scan trickles down into indexam.c, which now > has to be able to make decisions that are consistent with the ones > that the executor will make. That doesn't seem good at all. > I agree the all_visible flag is a sign the abstraction is not quite right. I did that mostly to quickly verify whether the duplicate VM checks are causing for the perf regression (and they are). Whatever the right abstraction is, it probably needs to do these VM checks only once. > * I think it might make sense to have two different prefetching > schemes. Ideally they could share some structure. If a caller is using > index_getnext_slot(), then it's easy for prefetching to be fully > transparent. The caller can just ask for TIDs and the prefetching > distance and TID queue can be fully under the control of something > that is hidden from the caller. But when using index_getnext_tid(), > the caller needs to have an opportunity to evaluate each TID and > decide whether we even want the heap tuple. If yes, then we feed that > TID to the prefetcher; if no, we don't. That way, we're not > replicating executor logic in lower-level code. However, that also > means that the IOS logic needs to be aware that this TID queue exists > and interact with whatever controls the prefetch distance. Perhaps > after calling index_getnext_tid() you call > index_prefetcher_put_tid(prefetcher, tid, bool fetch_heap_tuple) and > then you call index_prefetcher_get_tid() to drain the queue. Perhaps > also the prefetcher has a "fill" callback that gets invoked when the > TID queue isn't as full as the prefetcher wants it to be. Then > index_getnext_slot() can just install a trivial fill callback that > says index_prefetecher_put_tid(prefetcher, index_getnext_tid(...), > true), but IOS can use a more sophisticated callback that checks the > VM to determine what to pass for the third argument. > Yeah, after you pointed out the "leaky" abstraction, I also started to think about customizing the behavior using a callback. Not sure what exactly you mean by "fully transparent" but as I explained above I think we need to allow passing some information between the prefetcher and the executor - for example results of the visibility map checks in IOS. I have imagined something like this: nodeIndexscan / index_getnext_slot() -> no callback, all TIDs are prefetched nodeIndexonlyscan / index_getnext_tid() -> callback checks VM for the TID, prefetches if not all-visible -> the VM check result is stored in the queue with the VM (but in an extensible way, so that other callback can store other stuff) -> index_getnext_tid() also returns this extra information So not that different from the WIP patch, but in a "generic" and extensible way. Instead of hard-coding the all-visible flag, there'd be a something custom information. A bit like qsort_r() has a void* arg to pass custom context. Or if envisioned something different, could you elaborate a bit? > * I realize that I'm being a little inconsistent in what I just said, > because in the first bullet point I said that this wasn't really index > prefetching, and now I'm proposing function names that still start > with index_prefetch. It's not entirely clear to me what the best thing > to do about the terminology is here -- could it be a heap prefetcher, > or a TID prefetcher, or an index scan prefetcher? I don't really know, > but whatever we can do to make the naming more clear seems like a > really good idea. Maybe there should be a clearer separation between > the queue of TIDs that we're going to return from the index and the > queue of blocks that we want to prefetch to get the corresponding heap > tuples -- making that separation crisper might ease some of the naming > issues. > I think if the code stays in indexam.c, it's sensible to keep the index_ prefix, but then also have a more appropriate rest of the name. For example it might be index_prefetch_heap_pages() or something like that. > * Not that I want to be critical because I think this is a great start > on an important project, but it does look like there's an awful lot of > stuff here that still needs to be sorted out before it would be > reasonable to think of committing this, both in terms of design > decisions and just general polish. There's a lot of stuff marked with > XXX and I think that's great because most of those seem to be good > questions but that does leave the, err, small problem of figuring out > the answers. Absolutely. I certainly don't claim this is close to commit ... > index_prefetch_is_sequential() makes me really nervous > because it seems to depend an awful lot on whether the OS is doing > prefetching, and how the OS is doing prefetching, and I think those > might not be consistent across all systems and kernel versions. If the OS does not have read-ahead, or it's not configured properly, then the patch does not perform worse than what we have now. I'm far more concerned about the opposite issue, i.e. causing regressions with OS-level read-ahead. And the check handles that well, I think. > Similarly with index_prefetch(). There's a lot of "magical" > assumptions here. Even index_prefetch_add_cache() has this problem -- > the function assumes that it's OK if we sometimes fail to detect a > duplicate prefetch request, which makes sense, but under what > circumstances is it necessary to detect duplicates and in what cases > is it optional? The function comments are silent about that, which > makes it hard to assess whether the algorithm is good enough. > I don't quite understand what problem with duplicates you envision here. Strictly speaking, we don't need to detect/prevent duplicates - it's just that if you do posix_fadvise() for a block that's already in memory, it's overhead / wasted time. The whole point is to not do that very often. In this sense it's entirely optional, but desirable. I'm in no way claiming the comments are perfect, ofc. > * In terms of polish, one thing I noticed is that index_getnext_slot() > calls index_prefetch_tids() even when scan->xs_heap_continue is set, > which seems like it must be a waste, since we can't really need to > kick off more prefetch requests halfway through a HOT chain referenced > by a single index tuple, can we? Yeah, I think that's true. > Also, blks_prefetch_rounds doesn't > seem to be used anywhere, and neither that nor blks_prefetches are > documented. In fact there's no new documentation at all, which seems > probably not right. That's partly because there are no new GUCs, which > I feel like typically for a feature like this would be the place where > the feature behavior would be mentioned in the documentation. That's mostly because the explain fields were added to help during development. I'm not sure we actually want to make them part of EXPLAIN. > I don't > think it's a good idea to tie the behavior of this feature to > effective_io_concurrency partly because it's usually a bad idea to > make one setting control multiple different things, but perhaps even > more because effective_io_concurrency doesn't actually work in a > useful way AFAICT and people typically have to set it to some very > artificially large value compared to how much real I/O parallelism > they have. So probably there should be new GUCs with hopefully-better > semantics, but at least the documentation for any existing ones would > need updating, I would think. > I really don't want to have multiple knobs. At this point we have three GUCs, each tuning prefetching for a fairly large part of the system: effective_io_concurrency = regular queries maintenance_io_concurrency = utility commands recovery_prefetch = recovery / PITR This seems sensible, but I really don't want many more GUCs tuning prefetching for different executor nodes or something like that. If we have issues with how effective_io_concurrency works (and I'm not sure that's actually true), then perhaps we should fix that rather than inventing new GUCs. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-12-21 06:49 Dilip Kumar <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Dilip Kumar @ 2023-12-21 06:49 UTC (permalink / raw) To: Tomas Vondra <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> On Wed, Dec 20, 2023 at 7:11 AM Tomas Vondra <[email protected]> wrote: > I was going through to understand the idea, couple of observations -- + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) + { + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; + + /* Is this the oldest prefetch request in this LRU? */ + if (entry->request < oldestRequest) + { + oldestRequest = entry->request; + oldestIndex = i; + } + + /* + * If the entry is unused (identified by request being set to 0), + * we're done. Notice the field is uint64, so empty entry is + * guaranteed to be the oldest one. + */ + if (entry->request == 0) + continue; If the 'entry->request == 0' then we should break instead of continue, right? --- /* * Used to detect sequential patterns (and disable prefetching). */ #define PREFETCH_QUEUE_HISTORY 8 #define PREFETCH_SEQ_PATTERN_BLOCKS 4 If for sequential patterns we search only 4 blocks then why we are maintaining history for 8 blocks --- + * + * XXX Perhaps this should be tied to effective_io_concurrency somehow? + * + * XXX Could it be harmful that we read the queue backwards? Maybe memory + * prefetching works better for the forward direction? + */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) Correct, I think if we fetch this forward it will have an advantage with memory prefetching. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2023-12-21 12:48 Tomas Vondra <[email protected]> parent: Dilip Kumar <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Tomas Vondra @ 2023-12-21 12:48 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>; Georgios <[email protected]> On 12/21/23 07:49, Dilip Kumar wrote: > On Wed, Dec 20, 2023 at 7:11 AM Tomas Vondra > <[email protected]> wrote: >> > I was going through to understand the idea, couple of observations > > -- > + for (int i = 0; i < PREFETCH_LRU_SIZE; i++) > + { > + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i]; > + > + /* Is this the oldest prefetch request in this LRU? */ > + if (entry->request < oldestRequest) > + { > + oldestRequest = entry->request; > + oldestIndex = i; > + } > + > + /* > + * If the entry is unused (identified by request being set to 0), > + * we're done. Notice the field is uint64, so empty entry is > + * guaranteed to be the oldest one. > + */ > + if (entry->request == 0) > + continue; > > If the 'entry->request == 0' then we should break instead of continue, right? > Yes, I think that's true. The small LRU caches are accessed/filled linearly, so once we find an empty entry, all following entries are going to be empty too. I thought this shouldn't make any difference, because the LRUs are very small (only 8 entries, and I don't think we should make them larger). And it's going to go away once the cache gets full. But now that I think about it, maybe this could matter for small queries that only ever hit a couple rows. Hmmm, I'll have to check. Thanks for noticing this! > --- > /* > * Used to detect sequential patterns (and disable prefetching). > */ > #define PREFETCH_QUEUE_HISTORY 8 > #define PREFETCH_SEQ_PATTERN_BLOCKS 4 > > If for sequential patterns we search only 4 blocks then why we are > maintaining history for 8 blocks > > --- Right, I think there's no reason to keep these two separate constants. I believe this is a remnant from an earlier patch version which tried to do something smarter, but I ended up abandoning that. > > + * > + * XXX Perhaps this should be tied to effective_io_concurrency somehow? > + * > + * XXX Could it be harmful that we read the queue backwards? Maybe memory > + * prefetching works better for the forward direction? > + */ > + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) > > Correct, I think if we fetch this forward it will have an advantage > with memory prefetching. > OK, although we only really have a couple uint32 values, so it should be the same cacheline I guess. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2024-02-14 13:34 Tomas Vondra <[email protected]> parent: Tomas Vondra <[email protected]> 1 sibling, 1 reply; 10+ messages in thread From: Tomas Vondra @ 2024-02-14 13:34 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Melanie Plageman <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>; Georgios <[email protected]>; Thomas Munro <[email protected]>; Konstantin Knizhnik <[email protected]>; Dilip Kumar <[email protected]> On 2/13/24 20:54, Peter Geoghegan wrote: > On Tue, Feb 13, 2024 at 2:01 PM Tomas Vondra > <[email protected]> wrote: >> On 2/7/24 22:48, Melanie Plageman wrote: >> I admit I haven't thought about kill_prior_tuple until you pointed out. >> Yeah, prefetching separates (de-synchronizes) the two scans (index and >> heap) in a way that prevents this optimization. Or at least makes it >> much more complex :-( > > Another thing that argues against doing this is that we might not need > to visit any more B-Tree leaf pages when there is a LIMIT n involved. > We could end up scanning a whole extra leaf page (including all of its > tuples) for want of the ability to "push down" a LIMIT to the index AM > (that's not what happens right now, but it isn't really needed at all > right now). > I'm not quite sure I understand what is "this" that you argue against. Are you saying we should not separate the two scans? If yes, is there a better way to do this? The LIMIT problem is not very clear to me either. Yes, if we get close to the end of the leaf page, we may need to visit the next leaf page. But that's kinda the whole point of prefetching - reading stuff ahead, and reading too far ahead is an inherent risk. Isn't that a problem we have even without LIMIT? The prefetch distance ramp up is meant to limit the impact. > This property of index scans is fundamental to how index scans work. > Pinning an index page as an interlock against concurrently TID > recycling by VACUUM is directly described by the index API docs [1], > even (the docs actually use terms like "buffer pin" rather than > something more abstract sounding). I don't think that anything > affecting that behavior should be considered an implementation detail > of the nbtree index AM as such (nor any particular index AM). > Good point. > I think that it makes sense to put the index AM in control here -- > that almost follows from what I said about the index AM API. The index > AM already needs to be in control, in about the same way, to deal with > kill_prior_tuple (plus it helps with the LIMIT issue I described). > In control how? What would be the control flow - what part would be managed by the index AM? I initially did the prefetching entirely in each index AM, but it was suggested doing this in the executor would be better. So I gradually moved it to executor. But the idea to combine this with the streaming read API seems as a move from executor back to the lower levels ... and now you're suggesting to make the index AM responsible for this again. I'm not saying any of those layering options is wrong, but it's not clear to me which is the right one. > There doesn't necessarily need to be much code duplication to make > that work. Offhand I suspect it would be kind of similar to how > deletion of LP_DEAD-marked index tuples by non-nbtree index AMs gets > by with generic logic implemented by > index_compute_xid_horizon_for_tuples -- that's all that we need to > determine a snapshotConflictHorizon value for recovery conflict > purposes. Note that index_compute_xid_horizon_for_tuples() reads > *index* pages, despite not being aware of the caller's index AM and > index tuple format. > > (The only reason why nbtree needs a custom solution is because it has > posting list tuples to worry about, unlike GiST and unlike Hash, which > consistently use unadorned generic IndexTuple structs with heap TID > represented in the standard/generic way only. While these concepts > probably all originated in nbtree, they're still not nbtree > implementation details.) > I haven't looked at the details, but I agree the LP_DEAD deletion seems like a sensible inspiration. >>> Having disabled kill_prior_tuple is why the mvcc test fails. Perhaps >>> there is an easier way to fix this, as I don't think the mvcc test >>> failed on Tomas' version. >>> >> >> I kinda doubt it worked correctly, considering I simply ignored the >> optimization. It's far more likely it just worked by luck. > > The test that did fail will have only revealed that the > kill_prior_tuple wasn't operating as expected -- which isn't the same > thing as giving wrong answers. > Possible. But AFAIK it did fail for Melanie, and I don't have a very good explanation for the difference in behavior. > Note that there are various ways that concurrent TID recycling might > prevent _bt_killitems() from setting LP_DEAD bits. It's totally > unsurprising that breaking kill_prior_tuple in some way could be > missed. Andres wrote the MVCC test in question precisely because > certain aspects of kill_prior_tuple were broken for months without > anybody noticing. > > [1] https://www.postgresql.org/docs/devel/index-locking.html Yeah. There's clearly plenty of space for subtle issues. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: index prefetching @ 2024-02-14 18:21 Peter Geoghegan <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Peter Geoghegan @ 2024-02-14 18:21 UTC (permalink / raw) To: Tomas Vondra <[email protected]>; +Cc: Melanie Plageman <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>; Georgios <[email protected]>; Thomas Munro <[email protected]>; Konstantin Knizhnik <[email protected]>; Dilip Kumar <[email protected]> On Wed, Feb 14, 2024 at 8:34 AM Tomas Vondra <[email protected]> wrote: > > Another thing that argues against doing this is that we might not need > > to visit any more B-Tree leaf pages when there is a LIMIT n involved. > > We could end up scanning a whole extra leaf page (including all of its > > tuples) for want of the ability to "push down" a LIMIT to the index AM > > (that's not what happens right now, but it isn't really needed at all > > right now). > > > > I'm not quite sure I understand what is "this" that you argue against. > Are you saying we should not separate the two scans? If yes, is there a > better way to do this? What I'm concerned about is the difficulty and complexity of any design that requires revising "63.4. Index Locking Considerations", since that's pretty subtle stuff. In particular, if prefetching "de-synchronizes" (to use your term) the index leaf page level scan and the heap page scan, then we'll probably have to totally revise the basic API. Maybe that'll actually turn out to be the right thing to do -- it could just be the only thing that can unleash the full potential of prefetching. But I'm not aware of any evidence that points in that direction. Are you? (I might have just missed it.) > The LIMIT problem is not very clear to me either. Yes, if we get close > to the end of the leaf page, we may need to visit the next leaf page. > But that's kinda the whole point of prefetching - reading stuff ahead, > and reading too far ahead is an inherent risk. Isn't that a problem we > have even without LIMIT? The prefetch distance ramp up is meant to limit > the impact. Right now, the index AM doesn't know anything about LIMIT at all. That doesn't matter, since the index AM can only read/scan one full leaf page before returning control back to the executor proper. The executor proper can just shut down the whole index scan upon finding that we've already returned N tuples for a LIMIT N. We don't do prefetching right now, but we also don't risk reading a leaf page that'll just never be needed. Those two things are in tension, but I don't think that that's quite the same thing as the usual standard prefetching tension/problem. Here there is uncertainty about whether what we're prefetching will *ever* be required -- not uncertainty about when exactly it'll be required. (Perhaps this distinction doesn't mean much to you. I'm just telling you how I think about it, in case it helps move the discussion forward.) > > This property of index scans is fundamental to how index scans work. > > Pinning an index page as an interlock against concurrently TID > > recycling by VACUUM is directly described by the index API docs [1], > > even (the docs actually use terms like "buffer pin" rather than > > something more abstract sounding). I don't think that anything > > affecting that behavior should be considered an implementation detail > > of the nbtree index AM as such (nor any particular index AM). > > > > Good point. The main reason why the index AM docs require this interlock is because we need such an interlock to make non-MVCC snapshot scans safe. If you remove the interlock (the buffer pin interlock that protects against TID recycling by VACUUM), you can still avoid the same race condition by using an MVCC snapshot. This is why using an MVCC snapshot is a requirement for bitmap index scans. I believe that it's also a requirement for index-only scans, but the index AM docs don't spell that out. Another factor that complicates things here is mark/restore processing. The design for that has the idea of processing one page at a time baked-in. Kinda like with the kill_prior_tuple issue. It's certainly possible that you could figure out various workarounds for each of these issues (plus the kill_prior_tuple issue) with a prefetching design that "de-synchronizes" the index access and the heap access. But it might well be better to extend the existing design in a way that just avoids all these problems in the first place. Maybe "de-synchronization" really can pay for itself (because the benefits will outweigh these costs), but if you go that way then I'd really prefer it that way. > > I think that it makes sense to put the index AM in control here -- > > that almost follows from what I said about the index AM API. The index > > AM already needs to be in control, in about the same way, to deal with > > kill_prior_tuple (plus it helps with the LIMIT issue I described). > > > > In control how? What would be the control flow - what part would be > managed by the index AM? ISTM that prefetching for an index scan is about the index scan itself, first and foremost. The heap accesses are usually the dominant cost, of course, but sometimes the index leaf page accesses really do make up a significant fraction of the overall cost of the index scan. Especially with an expensive index qual. So if you just assume that the TIDs returned by the index scan are the only thing that matters, you might have a model that's basically correct on average, but is occasionally very wrong. That's one reason for "putting the index AM in control". As I said back in June, we should probably be marrying information from the index scan with information from the heap. This is something that is arguably a modularity violation. But it might just be that you really do need to take information from both places to consistently make the right trade-off. Perhaps the best arguments for "putting the index AM in control" only work when you go to fix the problems that "naive de-synchronization" creates. Thinking about that side of things some more might make "putting the index AM in control" seem more natural. Suppose, for example, you try to make a prefetching design based on "de-synchronization" work with kill_prior_tuple -- suppose you try to fix that problem. You're likely going to need to make some kind of trade-off that gets you most of the advantages that that approach offers (assuming that there really are significant advantages), while still retaining most of the advantages that we already get from kill_prior_tuple (basically we want to LP_DEAD-mark index tuples with almost or exactly the same consistency as we manage today). Maybe your approach involves tracking multiple LSNs for each prefetch-pending leaf page, or perhaps you hold on to a pin on some number of leaf pages instead (right now nbtree does both [1], which I go into more below). Either way, you're pushing stuff down into the index AM. Note that we already hang onto more than one pin at a time in rare cases involving mark/restore processing. For example, it can happen for a merge join that happens to involve an unlogged index, if the markpos and curpos are a certain way relative to the current leaf page (yeah, really). So putting stuff like that under the control of the index AM (while also applying basic information that comes from the heap) in order to fix the kill_prior_tuple issue is arguably something that has a kind of a precedent for us to follow. Even if you disagree with me here ("precedent" might be overstating it), perhaps you still get some general sense of why I have an inkling that putting prefetching in the index AM is the way to go. It's very hard to provide one really strong justification for all this, and I'm certainly not expecting you to just agree with me right away. I'm also not trying to impose any conditions on committing this patch. Thinking about this some more, "making kill_prior_tuple work with de-synchronization" is a bit of a misleading way of putting it. The way that you'd actually work around this is (at a very high level) *dynamically* making some kind of *trade-off* between synchronization and desynchronization. Up until now, we've been talking in terms of a strict dichotomy between the old index AM API design (index-page-at-a-time synchronization), and a "de-synchronizing" prefetching design that embraces the opposite extreme -- a design where we only think in terms of heap TIDs, and completely ignore anything that happens in the index structure (and consequently makes kill_prior_tuple ineffective). That now seems like a false dichotomy. > I initially did the prefetching entirely in each index AM, but it was > suggested doing this in the executor would be better. So I gradually > moved it to executor. But the idea to combine this with the streaming > read API seems as a move from executor back to the lower levels ... and > now you're suggesting to make the index AM responsible for this again. I did predict that there'd be lots of difficulties around the layering back in June. :-) > I'm not saying any of those layering options is wrong, but it's not > clear to me which is the right one. I don't claim to know what the right trade-off is myself. The fact that all of these things are in tension doesn't surprise me. It's just a hard problem. > Possible. But AFAIK it did fail for Melanie, and I don't have a very > good explanation for the difference in behavior. If you take a look at _bt_killitems(), you'll see that it actually has two fairly different strategies for avoiding TID recycling race condition issues, applied in each of two different cases: 1. Cases where we really have held onto a buffer pin, per the index AM API -- the "inde AM orthodox" approach. (The aforementioned issue with unlogged indexes exists because with an unlogged index we must use approach 1, per the nbtree README section [1]). 2. Cases where we drop the pin as an optimization (also per [1]), and now have to detect the possibility of concurrent modifications by VACUUM (that could have led to concurrent TID recycling). We conservatively do nothing (don't mark any index tuples LP_DEAD), unless the LSN is exactly the same as it was back when the page was scanned/read by _bt_readpage(). So some accidental detail with LSNs (like using or not using an unlogged index) could cause bugs in this area to "accidentally fail to fail". Since the nbtree index AM has its own optimizations here, which probably has a tendency to mask problems/bugs. (I sometimes use unlogged indexes for some of my nbtree related test cases, just to reduce certain kinds of variability, including variability in this area.) [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/access/nbtree/README;h=52e6... -- Peter Geoghegan ^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2024-02-14 18:21 UTC | newest] Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-07-14 20:31 Re: index prefetching Tomas Vondra <[email protected]> 2023-10-16 15:34 ` Tomas Vondra <[email protected]> 2023-11-24 16:25 ` Tomas Vondra <[email protected]> 2023-12-09 18:08 ` Tomas Vondra <[email protected]> 2023-12-18 21:00 ` Robert Haas <[email protected]> 2023-12-20 01:41 ` Tomas Vondra <[email protected]> 2023-12-21 06:49 ` Dilip Kumar <[email protected]> 2023-12-21 12:48 ` Tomas Vondra <[email protected]> 2024-02-14 13:34 ` Tomas Vondra <[email protected]> 2024-02-14 18:21 ` Peter Geoghegan <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox