public inbox for [email protected]help / color / mirror / Atom feed
Re: Eliminate redundant tuple visibility check in vacuum 70+ messages / 5 participants [nested] [flat]
* Re: Eliminate redundant tuple visibility check in vacuum @ 2023-08-31 22:29 Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2023-08-31 22:29 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Thu, Aug 31, 2023 at 2:03 PM Robert Haas <[email protected]> wrote: > > I have a few suggestions: > > - Rather than removing the rather large comment block at the top of > lazy_scan_prune() I'd like to see it rewritten to explain how we now > deal with the problem. I'd suggest leaving the first paragraph ("Prior > to...") just as it is and replace all the words following "The > approach we take now is" with a description of the approach that this > patch takes to the problem. Good idea. I've updated the comment. I also explain why this new approach works in the commit message and reference the commit which added the previous approach. > - I'm not a huge fan of the caller of heap_page_prune() having to know > how to initialize the PruneResult. Can heap_page_prune() itself do > that work, so that presult is an out parameter rather than an in-out > parameter? Or, if not, can it be moved to a new helper function, like > heap_page_prune_init(), rather than having that logic in 2+ places? Ah, yes. Now that it has two callers, and since it is exclusively an output parameter, it is quite ugly to initialize it in both callers. Fixed in the attached. > - int ndeleted, > - nnewlpdead; > - > - ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin, > - limited_ts, &nnewlpdead, NULL); > + int ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin, > + limited_ts, &presult, NULL); > > - I don't particularly like merging the declaration with the > assignment unless the call is narrow enough that we don't need a line > break in there, which is not the case here. I have changed this. > - I haven't thoroughly investigated yet, but I wonder if there are any > other places where comments need updating. As a general note, I find > it desirable for a function's header comment to mention whether any > pointer parameters are in parameters, out parameters, or in-out > parameters, and what the contract between caller and callee actually > is. I've investigated vacuumlazy.c and pruneheap.c and looked at the commit that added the retry loop (8523492d4e349) to see everywhere it added comments and don't see anywhere else that needs updating. I have updated lazy_scan_prune()'s function header comment to describe the nature of the in-out and output parameters and the contract. - Melanie Attachments: [text/x-patch] v2-0001-Rebrand-LVPagePruneState-as-PruneResult.patch (15.1K, ../../CAAKRu_aM43xZztW8=_Ot-zX=pNAm7c-WBULesUWjXr3v9k_2yw@mail.gmail.com/2-v2-0001-Rebrand-LVPagePruneState-as-PruneResult.patch) download | inline diff: From 6ea7a2aa5698e08ce67d47efd3dbfb54be30d9cb Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Thu, 3 Aug 2023 15:08:58 -0400 Subject: [PATCH v2 1/2] Rebrand LVPagePruneState as PruneResult With this rename, and relocation to heapam.h, we can use PruneResult as an output parameter for on-access pruning as well. It also makes it harder to confuse with PruneState. We'll be adding and removing PruneResult fields in future commits, but this rename and relocation is separate to make the diff easier to understand. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/vacuumlazy.c | 122 ++++++++++++--------------- src/include/access/heapam.h | 19 +++++ src/tools/pgindent/typedefs.list | 2 +- 3 files changed, 76 insertions(+), 67 deletions(-) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 6a41ee635d..5e0a7422bb 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -212,24 +212,6 @@ typedef struct LVRelState int64 missed_dead_tuples; /* # removable, but not removed */ } LVRelState; -/* - * State returned by lazy_scan_prune() - */ -typedef struct LVPagePruneState -{ - bool hastup; /* Page prevents rel truncation? */ - bool has_lpdead_items; /* includes existing LP_DEAD items */ - - /* - * State describes the proper VM bit states to set for the page following - * pruning and freezing. all_visible implies !has_lpdead_items, but don't - * trust all_frozen result unless all_visible is also set to true. - */ - bool all_visible; /* Every item visible to all? */ - bool all_frozen; /* provided all_visible is also true */ - TransactionId visibility_cutoff_xid; /* For recovery conflicts */ -} LVPagePruneState; - /* Struct for saving and restoring vacuum error information. */ typedef struct LVSavedErrInfo { @@ -250,7 +232,7 @@ static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, bool sharelock, Buffer vmbuffer); static void lazy_scan_prune(LVRelState *vacrel, Buffer buf, BlockNumber blkno, Page page, - LVPagePruneState *prunestate); + PruneResult *presult); static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf, BlockNumber blkno, Page page, bool *hastup, bool *recordfreespace); @@ -854,7 +836,7 @@ lazy_scan_heap(LVRelState *vacrel) Buffer buf; Page page; bool all_visible_according_to_vm; - LVPagePruneState prunestate; + PruneResult presult; if (blkno == next_unskippable_block) { @@ -1019,12 +1001,12 @@ lazy_scan_heap(LVRelState *vacrel) * were pruned some time earlier. Also considers freezing XIDs in the * tuple headers of remaining items with storage. */ - lazy_scan_prune(vacrel, buf, blkno, page, &prunestate); + lazy_scan_prune(vacrel, buf, blkno, page, &presult); - Assert(!prunestate.all_visible || !prunestate.has_lpdead_items); + Assert(!presult.all_visible || !presult.has_lpdead_items); /* Remember the location of the last page with nonremovable tuples */ - if (prunestate.hastup) + if (presult.hastup) vacrel->nonempty_pages = blkno + 1; if (vacrel->nindexes == 0) @@ -1037,7 +1019,7 @@ lazy_scan_heap(LVRelState *vacrel) * performed here can be thought of as the one-pass equivalent of * a call to lazy_vacuum(). */ - if (prunestate.has_lpdead_items) + if (presult.has_lpdead_items) { Size freespace; @@ -1064,8 +1046,8 @@ lazy_scan_heap(LVRelState *vacrel) * * Our call to lazy_vacuum_heap_page() will have considered if * it's possible to set all_visible/all_frozen independently - * of lazy_scan_prune(). Note that prunestate was invalidated - * by lazy_vacuum_heap_page() call. + * of lazy_scan_prune(). Note that prune result was + * invalidated by lazy_vacuum_heap_page() call. */ freespace = PageGetHeapFreeSpace(page); @@ -1078,23 +1060,23 @@ lazy_scan_heap(LVRelState *vacrel) * There was no call to lazy_vacuum_heap_page() because pruning * didn't encounter/create any LP_DEAD items that needed to be * vacuumed. Prune state has not been invalidated, so proceed - * with prunestate-driven visibility map and FSM steps (just like - * the two-pass strategy). + * with prune result-driven visibility map and FSM steps (just + * like the two-pass strategy). */ Assert(dead_items->num_items == 0); } /* * Handle setting visibility map bit based on information from the VM - * (as of last lazy_scan_skip() call), and from prunestate + * (as of last lazy_scan_skip() call), and from presult */ - if (!all_visible_according_to_vm && prunestate.all_visible) + if (!all_visible_according_to_vm && presult.all_visible) { uint8 flags = VISIBILITYMAP_ALL_VISIBLE; - if (prunestate.all_frozen) + if (presult.all_frozen) { - Assert(!TransactionIdIsValid(prunestate.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); flags |= VISIBILITYMAP_ALL_FROZEN; } @@ -1114,7 +1096,7 @@ lazy_scan_heap(LVRelState *vacrel) PageSetAllVisible(page); MarkBufferDirty(buf); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, prunestate.visibility_cutoff_xid, + vmbuffer, presult.visibility_cutoff_xid, flags); } @@ -1148,7 +1130,7 @@ lazy_scan_heap(LVRelState *vacrel) * There should never be LP_DEAD items on a page with PD_ALL_VISIBLE * set, however. */ - else if (prunestate.has_lpdead_items && PageIsAllVisible(page)) + else if (presult.has_lpdead_items && PageIsAllVisible(page)) { elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u", vacrel->relname, blkno); @@ -1161,16 +1143,16 @@ lazy_scan_heap(LVRelState *vacrel) /* * If the all-visible page is all-frozen but not marked as such yet, * mark it as all-frozen. Note that all_frozen is only valid if - * all_visible is true, so we must check both prunestate fields. + * all_visible is true, so we must check both presult fields. */ - else if (all_visible_according_to_vm && prunestate.all_visible && - prunestate.all_frozen && + else if (all_visible_according_to_vm && presult.all_visible && + presult.all_frozen && !VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer)) { /* * Avoid relying on all_visible_according_to_vm as a proxy for the * page-level PD_ALL_VISIBLE bit being set, since it might have - * become stale -- even when all_visible is set in prunestate + * become stale -- even when all_visible is set in presult */ if (!PageIsAllVisible(page)) { @@ -1185,7 +1167,7 @@ lazy_scan_heap(LVRelState *vacrel) * since a snapshotConflictHorizon sufficient to make everything * safe for REDO was logged when the page's tuples were frozen. */ - Assert(!TransactionIdIsValid(prunestate.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, vmbuffer, InvalidTransactionId, VISIBILITYMAP_ALL_VISIBLE | @@ -1196,7 +1178,7 @@ lazy_scan_heap(LVRelState *vacrel) * Final steps for block: drop cleanup lock, record free space in the * FSM */ - if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming) + if (presult.has_lpdead_items && vacrel->do_index_vacuuming) { /* * Wait until lazy_vacuum_heap_rel() to save free space. This @@ -1514,8 +1496,16 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, /* * lazy_scan_prune() -- lazy_scan_heap() pruning and freezing. * + * Prune and freeze a single page. * Caller must hold pin and buffer cleanup lock on the buffer. * + * vacrel is an in-out parameter lazy_scan_prune() references for relation-wide + * state and updates with information needed by lazy_scan_heap() to reap dead + * tuples, update statistics, and move the relfrozenxid forward. + * + * presult is an output parameter initialized and updated exclusively by + * lazy_scan_prune(). + * * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about * whether or not a tuple should be considered DEAD. This happened when an @@ -1536,7 +1526,7 @@ lazy_scan_prune(LVRelState *vacrel, Buffer buf, BlockNumber blkno, Page page, - LVPagePruneState *prunestate) + PruneResult *presult) { Relation rel = vacrel->rel; OffsetNumber offnum, @@ -1595,11 +1585,11 @@ retry: * Now scan the page to collect LP_DEAD items and check for tuples * requiring freezing among remaining tuples with storage */ - prunestate->hastup = false; - prunestate->has_lpdead_items = false; - prunestate->all_visible = true; - prunestate->all_frozen = true; - prunestate->visibility_cutoff_xid = InvalidTransactionId; + presult->hastup = false; + presult->has_lpdead_items = false; + presult->all_visible = true; + presult->all_frozen = true; + presult->visibility_cutoff_xid = InvalidTransactionId; for (offnum = FirstOffsetNumber; offnum <= maxoff; @@ -1621,7 +1611,7 @@ retry: if (ItemIdIsRedirected(itemid)) { /* page makes rel truncation unsafe */ - prunestate->hastup = true; + presult->hastup = true; continue; } @@ -1701,13 +1691,13 @@ retry: * asynchronously. See SetHintBits for more info. Check that * the tuple is hinted xmin-committed because of that. */ - if (prunestate->all_visible) + if (presult->all_visible) { TransactionId xmin; if (!HeapTupleHeaderXminCommitted(tuple.t_data)) { - prunestate->all_visible = false; + presult->all_visible = false; break; } @@ -1719,14 +1709,14 @@ retry: if (!TransactionIdPrecedes(xmin, vacrel->cutoffs.OldestXmin)) { - prunestate->all_visible = false; + presult->all_visible = false; break; } /* Track newest xmin on page. */ - if (TransactionIdFollows(xmin, prunestate->visibility_cutoff_xid) && + if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) && TransactionIdIsNormal(xmin)) - prunestate->visibility_cutoff_xid = xmin; + presult->visibility_cutoff_xid = xmin; } break; case HEAPTUPLE_RECENTLY_DEAD: @@ -1737,7 +1727,7 @@ retry: * pruning.) */ recently_dead_tuples++; - prunestate->all_visible = false; + presult->all_visible = false; break; case HEAPTUPLE_INSERT_IN_PROGRESS: @@ -1748,11 +1738,11 @@ retry: * results. This assumption is a bit shaky, but it is what * acquire_sample_rows() does, so be consistent. */ - prunestate->all_visible = false; + presult->all_visible = false; break; case HEAPTUPLE_DELETE_IN_PROGRESS: /* This is an expected case during concurrent vacuum */ - prunestate->all_visible = false; + presult->all_visible = false; /* * Count such rows as live. As above, we assume the deleting @@ -1766,7 +1756,7 @@ retry: break; } - prunestate->hastup = true; /* page makes rel truncation unsafe */ + presult->hastup = true; /* page makes rel truncation unsafe */ /* Tuple with storage -- consider need to freeze */ if (heap_prepare_freeze_tuple(tuple.t_data, &vacrel->cutoffs, &pagefrz, @@ -1782,7 +1772,7 @@ retry: * definitely cannot be set all-frozen in the visibility map later on */ if (!totally_frozen) - prunestate->all_frozen = false; + presult->all_frozen = false; } /* @@ -1800,7 +1790,7 @@ retry: * page all-frozen afterwards (might not happen until final heap pass). */ if (pagefrz.freeze_required || tuples_frozen == 0 || - (prunestate->all_visible && prunestate->all_frozen && + (presult->all_visible && presult->all_frozen && fpi_before != pgWalUsage.wal_fpi)) { /* @@ -1838,11 +1828,11 @@ retry: * once we're done with it. Otherwise we generate a conservative * cutoff by stepping back from OldestXmin. */ - if (prunestate->all_visible && prunestate->all_frozen) + if (presult->all_visible && presult->all_frozen) { /* Using same cutoff when setting VM is now unnecessary */ - snapshotConflictHorizon = prunestate->visibility_cutoff_xid; - prunestate->visibility_cutoff_xid = InvalidTransactionId; + snapshotConflictHorizon = presult->visibility_cutoff_xid; + presult->visibility_cutoff_xid = InvalidTransactionId; } else { @@ -1865,7 +1855,7 @@ retry: */ vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - prunestate->all_frozen = false; + presult->all_frozen = false; tuples_frozen = 0; /* avoid miscounts in instrumentation */ } @@ -1878,7 +1868,7 @@ retry: */ #ifdef USE_ASSERT_CHECKING /* Note that all_frozen value does not matter when !all_visible */ - if (prunestate->all_visible && lpdead_items == 0) + if (presult->all_visible && lpdead_items == 0) { TransactionId cutoff; bool all_frozen; @@ -1887,7 +1877,7 @@ retry: Assert(false); Assert(!TransactionIdIsValid(cutoff) || - cutoff == prunestate->visibility_cutoff_xid); + cutoff == presult->visibility_cutoff_xid); } #endif @@ -1900,7 +1890,7 @@ retry: ItemPointerData tmp; vacrel->lpdead_item_pages++; - prunestate->has_lpdead_items = true; + presult->has_lpdead_items = true; ItemPointerSetBlockNumber(&tmp, blkno); @@ -1925,7 +1915,7 @@ retry: * Now that freezing has been finalized, unset all_visible. It needs * to reflect the present state of things, as expected by our caller. */ - prunestate->all_visible = false; + presult->all_visible = false; } /* Finally, add page-local counts to whole-VACUUM counts */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index faf5026519..9a07828e5a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -191,6 +191,25 @@ typedef struct HeapPageFreeze } HeapPageFreeze; +/* + * State returned from pruning + */ +typedef struct PruneResult +{ + bool hastup; /* Page prevents rel truncation? */ + bool has_lpdead_items; /* includes existing LP_DEAD items */ + + /* + * State describes the proper VM bit states to set for the page following + * pruning and freezing. all_visible implies !has_lpdead_items, but don't + * trust all_frozen result unless all_visible is also set to true. + */ + bool all_visible; /* Every item visible to all? */ + bool all_frozen; /* provided all_visible is also true */ + TransactionId visibility_cutoff_xid; /* For recovery conflicts */ +} PruneResult; + + /* ---------------- * function prototypes for heap access method * diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 49a33c0387..a7a8d2acd1 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1394,7 +1394,6 @@ LPVOID LPWSTR LSEG LUID -LVPagePruneState LVRelState LVSavedErrInfo LWLock @@ -2152,6 +2151,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.37.2 [text/x-patch] v2-0002-Reuse-heap_page_prune-tuple-visibility-statuses.patch (15.0K, ../../CAAKRu_aM43xZztW8=_Ot-zX=pNAm7c-WBULesUWjXr3v9k_2yw@mail.gmail.com/3-v2-0002-Reuse-heap_page_prune-tuple-visibility-statuses.patch) download | inline diff: From ee66907ed8a3af73ec974347dadab433d1a0a80f Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Thu, 31 Aug 2023 18:15:35 -0400 Subject: [PATCH v2 2/2] Reuse heap_page_prune() tuple visibility statuses heap_page_prune() obtains the HTSV_Result (tuple visibility status) returned from HeapTupleSatisfiesVacuum() for every tuple on the page and stores them in an array. By making this array available to heap_page_prune()'s caller lazy_scan_prune(), we can avoid an additional call to HeapTupleSatisfiesVacuum() when freezing the tuples and recording LP_DEAD items for vacuum. This saves resources and eliminates the possibility that vacuuming corrupted data results in a hang due to endless retry looping. This replaces the retry mechanism introduced in 8523492d4 to handle cases in which a tuple's inserting transaction aborted between the visibility check in heap_page_prune() and lazy_scan_prune()'s call to HeapTupleSatisfiesVacuum() -- rendering it dead but without a dead line pointer. We can instead reuse the tuple's original visibility status, circumventing any disagreements. We pass the HTSV array from heap_page_prune() back to lazy_scan_prune() in PruneResult. It is convenient to move heap_page_prune()'s other output parameters into the PruneResult. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 66 +++++++++++++++------------- src/backend/access/heap/vacuumlazy.c | 54 +++++++---------------- src/include/access/heapam.h | 14 +++++- 3 files changed, 64 insertions(+), 70 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 47b9e20915..512f62e27d 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -65,16 +65,6 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; - - /* - * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. - * - * Same indexing as ->marked. - */ - int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneState; /* Local functions */ @@ -83,7 +73,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate); + PruneState *prstate, PruneResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); @@ -191,6 +181,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer) if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { + PruneResult presult; + /* OK, try to get exclusive buffer lock */ if (!ConditionalLockBufferForCleanup(buffer)) return; @@ -202,11 +194,10 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - int ndeleted, - nnewlpdead; + int ndeleted; ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin, - limited_ts, &nnewlpdead, NULL); + limited_ts, &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -222,9 +213,9 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * tracks ndeleted, since it will set the same LP_DEAD items to * LP_UNUSED separately. */ - if (ndeleted > nnewlpdead) + if (ndeleted > presult.nnewlpdead) pgstat_update_heap_dead_tuples(relation, - ndeleted - nnewlpdead); + ndeleted - presult.nnewlpdead); } /* And release buffer lock */ @@ -253,12 +244,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * either have been set by TransactionIdLimitedForOldSnapshots, or * InvalidTransactionId/0 respectively. * - * Sets *nnewlpdead for caller, indicating the number of items that were - * newly set LP_DEAD during prune operation. - * * off_loc is the offset location required by the caller to use in error * callback. * + * presult contains output parameters relevant to the caller. + * For example, sets presult->nnewlpdead for caller, indicating the number of + * items that were newly set LP_DEAD during prune operation. + * * Returns the number of tuples deleted from the page during this call. */ int @@ -266,7 +258,7 @@ heap_page_prune(Relation relation, Buffer buffer, GlobalVisState *vistest, TransactionId old_snap_xmin, TimestampTz old_snap_ts, - int *nnewlpdead, + PruneResult *presult, OffsetNumber *off_loc) { int ndeleted = 0; @@ -301,6 +293,19 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); + /* Initialize PruneResult */ + presult->hastup = false; + presult->has_lpdead_items = false; + presult->all_visible = true; + presult->all_frozen = true; + presult->visibility_cutoff_xid = InvalidTransactionId; + + /* + * nnewlpdead only includes those items which were newly set to LP_DEAD + * during pruning. + */ + presult->nnewlpdead = 0; + /* * Determine HTSV for all tuples. * @@ -331,7 +336,7 @@ heap_page_prune(Relation relation, Buffer buffer, /* Nothing to do if slot doesn't contain a tuple */ if (!ItemIdIsNormal(itemid)) { - prstate.htsv[offnum] = -1; + presult->htsv[offnum] = -1; continue; } @@ -347,8 +352,8 @@ heap_page_prune(Relation relation, Buffer buffer, if (off_loc) *off_loc = offnum; - prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, - buffer); + presult->htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, + buffer); } /* Scan the page */ @@ -372,7 +377,7 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - ndeleted += heap_prune_chain(buffer, offnum, &prstate); + ndeleted += heap_prune_chain(buffer, offnum, &prstate, presult); } /* Clear the offset information once we have processed the given page. */ @@ -473,7 +478,7 @@ heap_page_prune(Relation relation, Buffer buffer, END_CRIT_SECTION(); /* Record number of newly-set-LP_DEAD items for caller */ - *nnewlpdead = prstate.ndead; + presult->nnewlpdead = prstate.ndead; return ndeleted; } @@ -588,7 +593,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) * Returns the number of tuples (to be) deleted from the page. */ static int -heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) +heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + PruneState *prstate, PruneResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -609,7 +615,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ if (ItemIdIsNormal(rootlp)) { - Assert(prstate->htsv[rootoffnum] != -1); + Assert(presult->htsv[rootoffnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, rootlp); if (HeapTupleHeaderIsHeapOnly(htup)) @@ -632,7 +638,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) * either here or while following a chain below. Whichever path * gets there first will mark the tuple unused. */ - if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD && + if (presult->htsv[rootoffnum] == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup)) { heap_prune_record_unused(prstate, rootoffnum); @@ -700,7 +706,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) break; Assert(ItemIdIsNormal(lp)); - Assert(prstate->htsv[offnum] != -1); + Assert(presult->htsv[offnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, lp); /* @@ -720,7 +726,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ tupdead = recent_dead = false; - switch ((HTSV_Result) prstate->htsv[offnum]) + switch ((HTSV_Result) presult->htsv[offnum]) { case HEAPTUPLE_DEAD: tupdead = true; diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 5e0a7422bb..25ed500b45 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1504,7 +1504,7 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * tuples, update statistics, and move the relfrozenxid forward. * * presult is an output parameter initialized and updated exclusively by - * lazy_scan_prune(). + * lazy_scan_prune() and heap_page_prune(). * * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about @@ -1514,12 +1514,12 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * of complexity just so we could deal with tuples that were DEAD to VACUUM, * but nevertheless were left with storage after pruning. * - * The approach we take now is to restart pruning when the race condition is - * detected. This allows heap_page_prune() to prune the tuples inserted by - * the now-aborted transaction. This is a little crude, but it guarantees - * that any items that make it into the dead_items array are simple LP_DEAD - * line pointers, and that every remaining item with tuple storage is - * considered as a candidate for freezing. + * As of Postgres 17, we circumvent this problem altogether by resusing the + * result of heap_page_prune()'s visibility check. Without the second call to + * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no + * disagreement. The tuple's TID won't be added to the array of dead tuple TIDs + * for vacuum, thus vacuum will still never be tasked with reaping a tuple with + * storage. */ static void lazy_scan_prune(LVRelState *vacrel, @@ -1532,14 +1532,11 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - HeapTupleData tuple; - HTSV_Result res; int tuples_deleted, tuples_frozen, lpdead_items, live_tuples, recently_dead_tuples; - int nnewlpdead; HeapPageFreeze pagefrz; int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; @@ -1554,8 +1551,6 @@ lazy_scan_prune(LVRelState *vacrel, */ maxoff = PageGetMaxOffsetNumber(page); -retry: - /* Initialize (or reset) page-level state */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; @@ -1578,23 +1573,19 @@ retry: * that were deleted from indexes. */ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, - InvalidTransactionId, 0, &nnewlpdead, + InvalidTransactionId, 0, + presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples * requiring freezing among remaining tuples with storage */ - presult->hastup = false; - presult->has_lpdead_items = false; - presult->all_visible = true; - presult->all_frozen = true; - presult->visibility_cutoff_xid = InvalidTransactionId; - for (offnum = FirstOffsetNumber; offnum <= maxoff; offnum = OffsetNumberNext(offnum)) { + HeapTupleHeader htup; bool totally_frozen; /* @@ -1637,22 +1628,7 @@ retry: Assert(ItemIdIsNormal(itemid)); - ItemPointerSet(&(tuple.t_self), blkno, offnum); - tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid); - tuple.t_len = ItemIdGetLength(itemid); - tuple.t_tableOid = RelationGetRelid(rel); - - /* - * DEAD tuples are almost always pruned into LP_DEAD line pointers by - * heap_page_prune(), but it's possible that the tuple state changed - * since heap_page_prune() looked. Handle that here by restarting. - * (See comments at the top of function for a full explanation.) - */ - res = HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin, - buf); - - if (unlikely(res == HEAPTUPLE_DEAD)) - goto retry; + htup = (HeapTupleHeader) PageGetItem(page, itemid); /* * The criteria for counting a tuple as live in this block need to @@ -1673,7 +1649,7 @@ retry: * (Cases where we bypass index vacuuming will violate this optimistic * assumption, but the overall impact of that should be negligible.) */ - switch (res) + switch (presult->htsv[offnum]) { case HEAPTUPLE_LIVE: @@ -1695,7 +1671,7 @@ retry: { TransactionId xmin; - if (!HeapTupleHeaderXminCommitted(tuple.t_data)) + if (!HeapTupleHeaderXminCommitted(htup)) { presult->all_visible = false; break; @@ -1705,7 +1681,7 @@ retry: * The inserter definitely committed. But is it old enough * that everyone sees it as committed? */ - xmin = HeapTupleHeaderGetXmin(tuple.t_data); + xmin = HeapTupleHeaderGetXmin(htup); if (!TransactionIdPrecedes(xmin, vacrel->cutoffs.OldestXmin)) { @@ -1759,7 +1735,7 @@ retry: presult->hastup = true; /* page makes rel truncation unsafe */ /* Tuple with storage -- consider need to freeze */ - if (heap_prepare_freeze_tuple(tuple.t_data, &vacrel->cutoffs, &pagefrz, + if (heap_prepare_freeze_tuple(htup, &vacrel->cutoffs, &pagefrz, &frozen[tuples_frozen], &totally_frozen)) { /* Save prepared freeze plan for later */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 9a07828e5a..bf94bda634 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -199,6 +199,8 @@ typedef struct PruneResult bool hastup; /* Page prevents rel truncation? */ bool has_lpdead_items; /* includes existing LP_DEAD items */ + int nnewlpdead; /* Number of newly LP_DEAD items */ + /* * State describes the proper VM bit states to set for the page following * pruning and freezing. all_visible implies !has_lpdead_items, but don't @@ -207,6 +209,16 @@ typedef struct PruneResult bool all_visible; /* Every item visible to all? */ bool all_frozen; /* provided all_visible is also true */ TransactionId visibility_cutoff_xid; /* For recovery conflicts */ + + /* + * Tuple visibility is only computed once for each tuple, for correctness + * and efficiency reasons; see comment in heap_page_prune() for details. + * This is of type int8[], instead of HTSV_Result[], so we can use -1 to + * indicate no visibility has been computed, e.g. for LP_DEAD items. + * + * Same indexing as ->marked. + */ + int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneResult; @@ -307,7 +319,7 @@ extern int heap_page_prune(Relation relation, Buffer buffer, struct GlobalVisState *vistest, TransactionId old_snap_xmin, TimestampTz old_snap_ts, - int *nnewlpdead, + PruneResult *presult, OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, -- 2.37.2 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> @ 2023-09-06 17:04 ` Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Robert Haas @ 2023-09-06 17:04 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Thu, Aug 31, 2023 at 6:29 PM Melanie Plageman <[email protected]> wrote: > I have changed this. I spent a bunch of time today looking at this, thinking maybe I could commit it. But then I got cold feet. With these patches applied, PruneResult ends up being declared in heapam.h, with a comment that says /* State returned from pruning */. But that comment isn't accurate. The two new members that get added to the structure by 0002, namely nnewlpdead and htsv, are in fact state that is returned from pruning. But the other 5 members aren't. They're just initialized to constant values by pruning and then filled in for real by the vacuum logic. That's extremely weird. It would be fine if heap_page_prune() just grew a new output argument that only returned the HTSV results, or perhaps it could make sense to bundle any existing out parameters together into a struct and then add new things to that struct instead of adding even more parameters to the function itself. But there doesn't seem to be any good reason to muddle together the new output parameters for heap_page_prune() with a bunch of state that is currently internal to vacuumlazy.c. I realize that the shape of the patches probably stems from the fact that they started out life as part of a bigger patch set. But to be committed independently, they need to be shaped in a way that makes sense independently, and I don't think this qualifies. On the plus side, it seems to me that it's probably not that much work to fix this issue and that the result would likely be a smaller patch than what you have now, which is something. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> @ 2023-09-06 21:21 ` Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2023-09-06 21:21 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Wed, Sep 6, 2023 at 1:04 PM Robert Haas <[email protected]> wrote: > > On Thu, Aug 31, 2023 at 6:29 PM Melanie Plageman > <[email protected]> wrote: > > I have changed this. > > I spent a bunch of time today looking at this, thinking maybe I could > commit it. But then I got cold feet. > > With these patches applied, PruneResult ends up being declared in > heapam.h, with a comment that says /* State returned from pruning */. > But that comment isn't accurate. The two new members that get added to > the structure by 0002, namely nnewlpdead and htsv, are in fact state > that is returned from pruning. But the other 5 members aren't. They're > just initialized to constant values by pruning and then filled in for > real by the vacuum logic. That's extremely weird. It would be fine if > heap_page_prune() just grew a new output argument that only returned > the HTSV results, or perhaps it could make sense to bundle any > existing out parameters together into a struct and then add new things > to that struct instead of adding even more parameters to the function > itself. But there doesn't seem to be any good reason to muddle > together the new output parameters for heap_page_prune() with a bunch > of state that is currently internal to vacuumlazy.c. > > I realize that the shape of the patches probably stems from the fact > that they started out life as part of a bigger patch set. But to be > committed independently, they need to be shaped in a way that makes > sense independently, and I don't think this qualifies. On the plus > side, it seems to me that it's probably not that much work to fix this > issue and that the result would likely be a smaller patch than what > you have now, which is something. Yeah, I think this is a fair concern. I have addressed it in the attached patches. I thought a lot about whether or not adding a PruneResult which contains only the output parameters and result of heap_page_prune() is annoying since we have so many other *Prune* data structures. I decided it's not annoying. In some cases, four outputs don't merit a new structure. In this case, I think it declutters the code a bit -- independent of any other patches I may be writing :) - Melanie Attachments: [text/x-patch] v3-0001-Move-heap_page_prune-output-parameters-into-struc.patch (8.7K, ../../CAAKRu_a0kgBKgkJzDM0z=BGa34JDKiNC9WjoYcwXorKRXTnJMA@mail.gmail.com/2-v3-0001-Move-heap_page_prune-output-parameters-into-struc.patch) download | inline diff: From 11f5d895e4efe7b3b3a7bbc58efbb4d6c40274c0 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Wed, 6 Sep 2023 14:57:20 -0400 Subject: [PATCH v3 1/2] Move heap_page_prune output parameters into struct Add PruneResult, a structure containing the output parameters and result of heap_page_prune(). Reorganizing the results of heap_page_prune() into a struct simplifies the function signature and provides a location for future commits to store additional output parameters. Note that heap_page_prune() no longer NULL checks off_loc, the current tuple's offset in the line pointer array. It will never be NULL now that it is a member of a required output parameter, PruneResult. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 47 +++++++++++----------------- src/backend/access/heap/vacuumlazy.c | 19 +++++------ src/include/access/heapam.h | 20 ++++++++++-- src/tools/pgindent/typedefs.list | 1 + 4 files changed, 45 insertions(+), 42 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 18193efa23..f0847795a7 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -155,15 +155,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - int ndeleted, - nnewlpdead; + PruneResult presult; - ndeleted = heap_page_prune(relation, buffer, vistest, - &nnewlpdead, NULL); + heap_page_prune(relation, buffer, vistest, &presult); /* * Report the number of tuples reclaimed to pgstats. This is - * ndeleted minus the number of newly-LP_DEAD-set items. + * presult.ndeleted minus the number of newly-LP_DEAD-set items. * * We derive the number of dead tuples like this to avoid totally * forgetting about items that were set to LP_DEAD, since they @@ -175,9 +173,9 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * tracks ndeleted, since it will set the same LP_DEAD items to * LP_UNUSED separately. */ - if (ndeleted > nnewlpdead) + if (presult.ndeleted > presult.nnewlpdead) pgstat_update_heap_dead_tuples(relation, - ndeleted - nnewlpdead); + presult.ndeleted - presult.nnewlpdead); } /* And release buffer lock */ @@ -204,21 +202,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * (see heap_prune_satisfies_vacuum and * HeapTupleSatisfiesVacuum). * - * Sets *nnewlpdead for caller, indicating the number of items that were - * newly set LP_DEAD during prune operation. - * - * off_loc is the offset location required by the caller to use in error - * callback. - * - * Returns the number of tuples deleted from the page during this call. + * presult contains output parameters needed by callers such as the number of + * tuples removed and the number of line pointers newly marked LP_DEAD. + * heap_page_prune() is responsible for initializing it. */ -int +void heap_page_prune(Relation relation, Buffer buffer, GlobalVisState *vistest, - int *nnewlpdead, - OffsetNumber *off_loc) + PruneResult *presult) { - int ndeleted = 0; Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); OffsetNumber offnum, @@ -244,6 +236,10 @@ heap_page_prune(Relation relation, Buffer buffer, prstate.nredirected = prstate.ndead = prstate.nunused = 0; memset(prstate.marked, 0, sizeof(prstate.marked)); + presult->ndeleted = 0; + presult->nnewlpdead = 0; + presult->off_loc = InvalidOffsetNumber; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -290,8 +286,7 @@ heap_page_prune(Relation relation, Buffer buffer, * Set the offset number so that we can display it along with any * error that occurred while processing this tuple. */ - if (off_loc) - *off_loc = offnum; + presult->off_loc = offnum; prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, buffer); @@ -309,8 +304,7 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* see preceding loop */ - if (off_loc) - *off_loc = offnum; + presult->off_loc = offnum; /* Nothing to do if slot is empty or already dead */ itemid = PageGetItemId(page, offnum); @@ -318,12 +312,11 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - ndeleted += heap_prune_chain(buffer, offnum, &prstate); + presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); } /* Clear the offset information once we have processed the given page. */ - if (off_loc) - *off_loc = InvalidOffsetNumber; + presult->off_loc = InvalidOffsetNumber; /* Any error while applying the changes is critical */ START_CRIT_SECTION(); @@ -419,9 +412,7 @@ heap_page_prune(Relation relation, Buffer buffer, END_CRIT_SECTION(); /* Record number of newly-set-LP_DEAD items for caller */ - *nnewlpdead = prstate.ndead; - - return ndeleted; + presult->nnewlpdead = prstate.ndead; } diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 1a05adfa61..a3c756ab2e 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1544,12 +1544,11 @@ lazy_scan_prune(LVRelState *vacrel, ItemId itemid; HeapTupleData tuple; HTSV_Result res; - int tuples_deleted, - tuples_frozen, + PruneResult presult; + int tuples_frozen, lpdead_items, live_tuples, recently_dead_tuples; - int nnewlpdead; HeapPageFreeze pagefrz; int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; @@ -1572,7 +1571,6 @@ retry: pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid; - tuples_deleted = 0; tuples_frozen = 0; lpdead_items = 0; live_tuples = 0; @@ -1581,15 +1579,14 @@ retry: /* * Prune all HOT-update chains in this page. * - * We count tuples removed by the pruning step as tuples_deleted. Its - * final value can be thought of as the number of tuples that have been - * deleted from the table. It should not be confused with lpdead_items; + * We count the number of tuples removed from the page by the pruning step + * in presult.ndeleted. It should not be confused with lpdead_items; * lpdead_items's final value can be thought of as the number of tuples * that were deleted from indexes. */ - tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, - &nnewlpdead, - &vacrel->offnum); + heap_page_prune(rel, buf, vacrel->vistest, &presult); + + vacrel->offnum = presult.off_loc; /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1929,7 +1926,7 @@ retry: } /* Finally, add page-local counts to whole-VACUUM counts */ - vacrel->tuples_deleted += tuples_deleted; + vacrel->tuples_deleted += presult.ndeleted; vacrel->tuples_frozen += tuples_frozen; vacrel->lpdead_items += lpdead_items; vacrel->live_tuples += live_tuples; diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 6598c4d7d8..a2ea3be52b 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -191,6 +191,21 @@ typedef struct HeapPageFreeze } HeapPageFreeze; +/* + * Per-page state returned from pruning + */ +typedef struct PruneResult +{ + int ndeleted; /* Number of tuples deleted from the page */ + int nnewlpdead; /* Number of newly LP_DEAD items */ + + /* + * Current tuple's offset in the line pointer array, used for error + * callback. + */ + OffsetNumber off_loc; +} PruneResult; + /* ---------------- * function prototypes for heap access method * @@ -284,10 +299,9 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern int heap_page_prune(Relation relation, Buffer buffer, +extern void heap_page_prune(Relation relation, Buffer buffer, struct GlobalVisState *vistest, - int *nnewlpdead, - OffsetNumber *off_loc); + PruneResult *presult); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 0656c94416..8c30c0b1b6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2150,6 +2150,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.37.2 [text/x-patch] v3-0002-Reuse-heap_page_prune-tuple-visibility-statuses.patch (10.8K, ../../CAAKRu_a0kgBKgkJzDM0z=BGa34JDKiNC9WjoYcwXorKRXTnJMA@mail.gmail.com/3-v3-0002-Reuse-heap_page_prune-tuple-visibility-statuses.patch) download | inline diff: From a02e108f9637443251f1b9d48fadbf339f710408 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Wed, 6 Sep 2023 16:54:41 -0400 Subject: [PATCH v3 2/2] Reuse heap_page_prune() tuple visibility statuses heap_page_prune() obtains the HTSV_Result (tuple visibility status) returned from HeapTupleSatisfiesVacuum() for every tuple on the page and stores them in an array. By making this array available to heap_page_prune()'s caller lazy_scan_prune(), we can avoid an additional call to HeapTupleSatisfiesVacuum() when freezing the tuples and recording LP_DEAD items for vacuum. This saves resources and eliminates the possibility that vacuuming corrupted data results in a hang due to endless retry looping. This replaces the retry mechanism introduced in 8523492d4 to handle cases in which a tuple's inserting transaction aborted between the visibility check in heap_page_prune() and lazy_scan_prune()'s call to HeapTupleSatisfiesVacuum() -- rendering it dead but without a dead line pointer. We can instead reuse the tuple's original visibility status, circumventing any disagreements. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 37 ++++++++++++------------ src/backend/access/heap/vacuumlazy.c | 42 ++++++++-------------------- src/include/access/heapam.h | 11 ++++++++ 3 files changed, 41 insertions(+), 49 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index f0847795a7..2ca10b6dcf 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -53,16 +53,6 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; - - /* - * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. - * - * Same indexing as ->marked. - */ - int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneState; /* Local functions */ @@ -71,6 +61,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + int8 *htsv, PruneState *prstate); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, @@ -236,6 +227,10 @@ heap_page_prune(Relation relation, Buffer buffer, prstate.nredirected = prstate.ndead = prstate.nunused = 0; memset(prstate.marked, 0, sizeof(prstate.marked)); + /* + * presult->htsv is not initialized here because all ntuple spots in the + * array will be set either to a valid HTSV_Result value or -1. + */ presult->ndeleted = 0; presult->nnewlpdead = 0; presult->off_loc = InvalidOffsetNumber; @@ -273,7 +268,7 @@ heap_page_prune(Relation relation, Buffer buffer, /* Nothing to do if slot doesn't contain a tuple */ if (!ItemIdIsNormal(itemid)) { - prstate.htsv[offnum] = -1; + presult->htsv[offnum] = -1; continue; } @@ -288,8 +283,8 @@ heap_page_prune(Relation relation, Buffer buffer, */ presult->off_loc = offnum; - prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, - buffer); + presult->htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, + buffer); } /* Scan the page */ @@ -312,7 +307,8 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); + presult->ndeleted += heap_prune_chain(buffer, offnum, + presult->htsv, &prstate); } /* Clear the offset information once we have processed the given page. */ @@ -440,6 +436,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) /* * Prune specified line pointer or a HOT chain originating at line pointer. * + * Tuple visibility information is provided in htsv. + * * If the item is an index-referenced tuple (i.e. not a heap-only tuple), * the HOT chain is pruned by removing all DEAD tuples at the start of the HOT * chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple. @@ -467,7 +465,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) * Returns the number of tuples (to be) deleted from the page. */ static int -heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) +heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + int8 *htsv, PruneState *prstate) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -488,7 +487,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ if (ItemIdIsNormal(rootlp)) { - Assert(prstate->htsv[rootoffnum] != -1); + Assert(htsv[rootoffnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, rootlp); if (HeapTupleHeaderIsHeapOnly(htup)) @@ -511,7 +510,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) * either here or while following a chain below. Whichever path * gets there first will mark the tuple unused. */ - if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD && + if (htsv[rootoffnum] == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup)) { heap_prune_record_unused(prstate, rootoffnum); @@ -579,7 +578,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) break; Assert(ItemIdIsNormal(lp)); - Assert(prstate->htsv[offnum] != -1); + Assert(htsv[offnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, lp); /* @@ -599,7 +598,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ tupdead = recent_dead = false; - switch ((HTSV_Result) prstate->htsv[offnum]) + switch ((HTSV_Result) htsv[offnum]) { case HEAPTUPLE_DEAD: tupdead = true; diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index a3c756ab2e..ae813483a5 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1524,12 +1524,12 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * of complexity just so we could deal with tuples that were DEAD to VACUUM, * but nevertheless were left with storage after pruning. * - * The approach we take now is to restart pruning when the race condition is - * detected. This allows heap_page_prune() to prune the tuples inserted by - * the now-aborted transaction. This is a little crude, but it guarantees - * that any items that make it into the dead_items array are simple LP_DEAD - * line pointers, and that every remaining item with tuple storage is - * considered as a candidate for freezing. + * As of Postgres 17, we circumvent this problem altogether by reusing the + * result of heap_page_prune()'s visibility check. Without the second call to + * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no + * disagreement. The tuple's TID won't be added to the array of dead tuple TIDs + * for vacuum, thus vacuum will never be tasked with reaping a tuple with + * storage. */ static void lazy_scan_prune(LVRelState *vacrel, @@ -1542,8 +1542,6 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - HeapTupleData tuple; - HTSV_Result res; PruneResult presult; int tuples_frozen, lpdead_items, @@ -1563,8 +1561,6 @@ lazy_scan_prune(LVRelState *vacrel, */ maxoff = PageGetMaxOffsetNumber(page); -retry: - /* Initialize (or reset) page-level state */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; @@ -1602,6 +1598,7 @@ retry: offnum <= maxoff; offnum = OffsetNumberNext(offnum)) { + HeapTupleHeader htup; bool totally_frozen; /* @@ -1644,22 +1641,7 @@ retry: Assert(ItemIdIsNormal(itemid)); - ItemPointerSet(&(tuple.t_self), blkno, offnum); - tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid); - tuple.t_len = ItemIdGetLength(itemid); - tuple.t_tableOid = RelationGetRelid(rel); - - /* - * DEAD tuples are almost always pruned into LP_DEAD line pointers by - * heap_page_prune(), but it's possible that the tuple state changed - * since heap_page_prune() looked. Handle that here by restarting. - * (See comments at the top of function for a full explanation.) - */ - res = HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin, - buf); - - if (unlikely(res == HEAPTUPLE_DEAD)) - goto retry; + htup = (HeapTupleHeader) PageGetItem(page, itemid); /* * The criteria for counting a tuple as live in this block need to @@ -1680,7 +1662,7 @@ retry: * (Cases where we bypass index vacuuming will violate this optimistic * assumption, but the overall impact of that should be negligible.) */ - switch (res) + switch ((HTSV_Result) presult.htsv[offnum]) { case HEAPTUPLE_LIVE: @@ -1702,7 +1684,7 @@ retry: { TransactionId xmin; - if (!HeapTupleHeaderXminCommitted(tuple.t_data)) + if (!HeapTupleHeaderXminCommitted(htup)) { prunestate->all_visible = false; break; @@ -1712,7 +1694,7 @@ retry: * The inserter definitely committed. But is it old enough * that everyone sees it as committed? */ - xmin = HeapTupleHeaderGetXmin(tuple.t_data); + xmin = HeapTupleHeaderGetXmin(htup); if (!TransactionIdPrecedes(xmin, vacrel->cutoffs.OldestXmin)) { @@ -1766,7 +1748,7 @@ retry: prunestate->hastup = true; /* page makes rel truncation unsafe */ /* Tuple with storage -- consider need to freeze */ - if (heap_prepare_freeze_tuple(tuple.t_data, &vacrel->cutoffs, &pagefrz, + if (heap_prepare_freeze_tuple(htup, &vacrel->cutoffs, &pagefrz, &frozen[tuples_frozen], &totally_frozen)) { /* Save prepared freeze plan for later */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index a2ea3be52b..1fe8354975 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -204,6 +204,17 @@ typedef struct PruneResult * callback. */ OffsetNumber off_loc; + + /* + * Tuple visibility is only computed once for each tuple, for correctness + * and efficiency reasons; see comment in heap_page_prune() for details. + * This is of type int8[], instead of HTSV_Result[], so we can use -1 to + * indicate no visibility has been computed, e.g. for LP_DEAD items. + * + * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is + * 1. Otherwise every access would need to subtract 1. + */ + int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneResult; /* ---------------- -- 2.37.2 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> @ 2023-09-07 17:37 ` Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Robert Haas @ 2023-09-07 17:37 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Wed, Sep 6, 2023 at 5:21 PM Melanie Plageman <[email protected]> wrote: > Yeah, I think this is a fair concern. I have addressed it in the > attached patches. > > I thought a lot about whether or not adding a PruneResult which > contains only the output parameters and result of heap_page_prune() is > annoying since we have so many other *Prune* data structures. I > decided it's not annoying. In some cases, four outputs don't merit a > new structure. In this case, I think it declutters the code a bit -- > independent of any other patches I may be writing :) I took a look at 0001 and I think that it's incorrect. In the existing code, *off_loc gets updated before each call to heap_prune_satisfies_vacuum(). This means that if an error occurs in heap_prune_satisfies_vacuum(), *off_loc will as of that moment contain the relevant offset number. In your version, the relevant offset number will only be stored in some local structure to which the caller doesn't yet have access. The difference is meaningful. lazy_scan_prune passes off_loc as vacrel->offnum, which means that if an error happens, vacrel->offnum will have the right value, and so when the error context callback installed by heap_vacuum_rel() fires, namely vacuum_error_callback(), it can look at vacrel->offnum and know where the error happened. With your patch, I think that would no longer work. I haven't run the regression suite with 0001 applied. I'm guessing that you did, and that they passed, which if true means that we don't have a test for this, which is a shame, although writing such a test might be a bit tricky. If there is a test case for this and you didn't run it, woops. This is also why I think it's *extremely* important for the header comment of a function that takes pointer parameters to document the semantics of those pointers. Normally they are in parameters or out parameters or in-out parameters, but here it's something even more complicated. The existing header comment says "off_loc is the offset location required by the caller to use in error callback," which I didn't really understand until I actually looked at what the code is doing, so I consider that somebody could've done a better job writing this comment, but in theory you could've also noticed that, at least AFAICS, there's no way for the function to return with *off_loc set to anything other than InvalidOffsetNumber. That means that the statements which set *off_loc to other values must have some other purpose. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> @ 2023-09-07 19:09 ` Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2023-09-07 19:09 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Thu, Sep 7, 2023 at 1:37 PM Robert Haas <[email protected]> wrote: > > On Wed, Sep 6, 2023 at 5:21 PM Melanie Plageman > <[email protected]> wrote: > > Yeah, I think this is a fair concern. I have addressed it in the > > attached patches. > > > > I thought a lot about whether or not adding a PruneResult which > > contains only the output parameters and result of heap_page_prune() is > > annoying since we have so many other *Prune* data structures. I > > decided it's not annoying. In some cases, four outputs don't merit a > > new structure. In this case, I think it declutters the code a bit -- > > independent of any other patches I may be writing :) > > I took a look at 0001 and I think that it's incorrect. In the existing > code, *off_loc gets updated before each call to > heap_prune_satisfies_vacuum(). This means that if an error occurs in > heap_prune_satisfies_vacuum(), *off_loc will as of that moment contain > the relevant offset number. In your version, the relevant offset > number will only be stored in some local structure to which the caller > doesn't yet have access. The difference is meaningful. lazy_scan_prune > passes off_loc as vacrel->offnum, which means that if an error > happens, vacrel->offnum will have the right value, and so when the > error context callback installed by heap_vacuum_rel() fires, namely > vacuum_error_callback(), it can look at vacrel->offnum and know where > the error happened. With your patch, I think that would no longer > work. You are right. That is a major problem. Thank you for finding that. I was able to confirm the breakage by patching in an error to heap_page_prune() after we have set off_loc and confirming that the error context has the offset in master and is missing it with my patch applied. I can fix it by changing the type of PruneResult->off_loc to be an OffsetNumber pointer. This does mean that PruneResult will be initialized partially by heap_page_prune() callers. I wonder if you think that undermines the case for making a new struct. I still want to eliminate the NULL check of off_loc in heap_page_prune() by making it a required parameter. Even though on-access pruning does not have an error callback mechanism which uses the offset, it seems better to have a pointless local variable in heap_page_prune_opt() than to do a NULL check for every tuple pruned. > I haven't run the regression suite with 0001 applied. I'm guessing > that you did, and that they passed, which if true means that we don't > have a test for this, which is a shame, although writing such a test > might be a bit tricky. If there is a test case for this and you didn't > run it, woops. There is no test coverage for the vacuum error callback context currently (tests passed for me). I looked into how we might add such a test. First, I investigated what kind of errors might occur during heap_prune_satisfies_vacuum(). Some of the multixact functions called by HeapTupleSatisfiesVacuumHorizon() could error out -- for example GetMultiXactIdMembers(). It seems difficult to trigger the errors in GetMultiXactIdMembers(), as we would have to cause wraparound. It would be even more difficult to ensure that we hit those specific errors from a call stack containing heap_prune_satisfies_vacuum(). As such, I'm not sure I can think of a way to protect future developers from repeating my mistake--apart from improving the comment like you mentioned. - Melanie ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> @ 2023-09-07 19:29 ` Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Robert Haas @ 2023-09-07 19:29 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Thu, Sep 7, 2023 at 3:10 PM Melanie Plageman <[email protected]> wrote: > I can fix it by changing the type of PruneResult->off_loc to be an > OffsetNumber pointer. This does mean that PruneResult will be > initialized partially by heap_page_prune() callers. I wonder if you > think that undermines the case for making a new struct. I think that it undermines the case for including that particular argument in the struct. It's not really a Prune*Result* if the caller initializes it in part. It seems fairly reasonable to still have a PruneResult struct for the other stuff, though, at least to me. How do you see it? (One could also argue that this is a somewhat more byzantine way of doing error reporting than would be desirable, but fixing that problem doesn't seem too straightforward so perhaps it's prudent to leave it well enough alone.) > I still want to eliminate the NULL check of off_loc in > heap_page_prune() by making it a required parameter. Even though > on-access pruning does not have an error callback mechanism which uses > the offset, it seems better to have a pointless local variable in > heap_page_prune_opt() than to do a NULL check for every tuple pruned. It doesn't seem important to me unless it improves performance. If it's just stylistic, I don't object, but I also don't really see a reason to care. > > I haven't run the regression suite with 0001 applied. I'm guessing > > that you did, and that they passed, which if true means that we don't > > have a test for this, which is a shame, although writing such a test > > might be a bit tricky. If there is a test case for this and you didn't > > run it, woops. > > There is no test coverage for the vacuum error callback context > currently (tests passed for me). I looked into how we might add such a > test. First, I investigated what kind of errors might occur during > heap_prune_satisfies_vacuum(). Some of the multixact functions called > by HeapTupleSatisfiesVacuumHorizon() could error out -- for example > GetMultiXactIdMembers(). It seems difficult to trigger the errors in > GetMultiXactIdMembers(), as we would have to cause wraparound. It > would be even more difficult to ensure that we hit those specific > errors from a call stack containing heap_prune_satisfies_vacuum(). As > such, I'm not sure I can think of a way to protect future developers > from repeating my mistake--apart from improving the comment like you > mentioned. 004_verify_heapam.pl has some tests that intentionally corrupt pages and then use pg_amcheck to detect the corruption. Such an approach could probably also be used here. But it's a pain to get such tests right, because any change to the page format due to endianness, different block size, or whatever can make carefully-written tests go boom. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> @ 2023-09-07 22:23 ` Melanie Plageman <[email protected]> 2023-09-08 15:06 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-13 17:29 ` Re: Eliminate redundant tuple visibility check in vacuum Andres Freund <[email protected]> 0 siblings, 2 replies; 70+ messages in thread From: Melanie Plageman @ 2023-09-07 22:23 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Thu, Sep 7, 2023 at 3:30 PM Robert Haas <[email protected]> wrote: > > On Thu, Sep 7, 2023 at 3:10 PM Melanie Plageman > <[email protected]> wrote: > > I can fix it by changing the type of PruneResult->off_loc to be an > > OffsetNumber pointer. This does mean that PruneResult will be > > initialized partially by heap_page_prune() callers. I wonder if you > > think that undermines the case for making a new struct. > > I think that it undermines the case for including that particular > argument in the struct. It's not really a Prune*Result* if the caller > initializes it in part. It seems fairly reasonable to still have a > PruneResult struct for the other stuff, though, at least to me. How do > you see it? Yes, I think off_loc probably didn't belong in PruneResult to begin with. It is inherently not a result of pruning since it will only be used in the event that pruning doesn't complete (it errors out). In the attached v4 patch set, I have both PruneResult and off_loc as parameters to heap_page_prune(). I have also added a separate commit which adds comments both above heap_page_prune()'s call site in lazy_scan_prune() and in the heap_page_prune() function header clarifying the various points we discussed. > > I still want to eliminate the NULL check of off_loc in > > heap_page_prune() by making it a required parameter. Even though > > on-access pruning does not have an error callback mechanism which uses > > the offset, it seems better to have a pointless local variable in > > heap_page_prune_opt() than to do a NULL check for every tuple pruned. > > It doesn't seem important to me unless it improves performance. If > it's just stylistic, I don't object, but I also don't really see a > reason to care. I did some performance testing but, as expected, I couldn't concoct a scenario where the overhead was noticeable in a profile. So much else is happening in that code, the NULL check basically doesn't matter (even though it isn't optimized out). I mostly wanted to remove the NULL checks because I found them distracting (so, a stylistic complaint). However, upon further reflection, I actually think it is better if heap_page_prune_opt() passes NULL. heap_page_prune() has no error callback mechanism that could use it, and passing a valid value implies otherwise. Also, as you said, off_loc will always be InvalidOffsetNumber when heap_page_prune() returns normally, so having heap_page_prune_opt() pass a dummy value might actually be more confusing for future programmers. > > > I haven't run the regression suite with 0001 applied. I'm guessing > > > that you did, and that they passed, which if true means that we don't > > > have a test for this, which is a shame, although writing such a test > > > might be a bit tricky. If there is a test case for this and you didn't > > > run it, woops. > > > > There is no test coverage for the vacuum error callback context > > currently (tests passed for me). I looked into how we might add such a > > test. First, I investigated what kind of errors might occur during > > heap_prune_satisfies_vacuum(). Some of the multixact functions called > > by HeapTupleSatisfiesVacuumHorizon() could error out -- for example > > GetMultiXactIdMembers(). It seems difficult to trigger the errors in > > GetMultiXactIdMembers(), as we would have to cause wraparound. It > > would be even more difficult to ensure that we hit those specific > > errors from a call stack containing heap_prune_satisfies_vacuum(). As > > such, I'm not sure I can think of a way to protect future developers > > from repeating my mistake--apart from improving the comment like you > > mentioned. > > 004_verify_heapam.pl has some tests that intentionally corrupt pages > and then use pg_amcheck to detect the corruption. Such an approach > could probably also be used here. But it's a pain to get such tests > right, because any change to the page format due to endianness, > different block size, or whatever can make carefully-written tests go > boom. Cool! I hadn't examined how these tests work until now. I took a stab at writing a test in the existing 0004_verify_heapam.pl. The simplest thing would be if we could just vacuum the corrupted table ("test") after running pg_amcheck and compare the error context to our expectation. I found that this didn't work, though. In an assert build, vacuum trips an assert before it hits an error while vacuuming a corrupted tuple in the "test" table. There might be a way of modifying the existing test code to avoid this, but I tried the next easiest thing -- corrupting a tuple in the other existing table in the file, "junk". This is possible to do, but we have to repeat a lot of the setup code done for the "test" table to get the line pointer array and loop through and corrupt a tuple. In order to do this well, I would want to refactor some of the boilerplate into a function. There are other fiddly bits as well that I needed to consider. I think a test like this could be useful coverage of the some of the possible errors that could happen in heap_prune_satisfies_vacuum(), but it definitely isn't coverage of pg_amcheck (and thus shouldn't go in that file) and a whole new test which spins up a Postgres to cover vacuum_error_callback() seemed like a bit much. - Melanie Attachments: [text/x-patch] v4-0002-Move-heap_page_prune-output-parameters-into-struc.patch (7.9K, ../../CAAKRu_YRv4OvT3t4Dw+8UCuji2aVmQxmkJpXvqvTBTWBHEJ-ww@mail.gmail.com/2-v4-0002-Move-heap_page_prune-output-parameters-into-struc.patch) download | inline diff: From e986940e546171d1f1d06f62a101d695a8481e7a Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Wed, 6 Sep 2023 14:57:20 -0400 Subject: [PATCH v4 2/3] Move heap_page_prune output parameters into struct Add PruneResult, a structure containing the output parameters and result of heap_page_prune(). Reorganizing the results of heap_page_prune() into a struct simplifies the function signature and provides a location for future commits to store additional output parameters. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 33 +++++++++++++--------------- src/backend/access/heap/vacuumlazy.c | 17 +++++--------- src/include/access/heapam.h | 13 +++++++++-- src/tools/pgindent/typedefs.list | 1 + 4 files changed, 33 insertions(+), 31 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 392b54f093..5ac286e152 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -155,15 +155,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - int ndeleted, - nnewlpdead; + PruneResult presult; - ndeleted = heap_page_prune(relation, buffer, vistest, - &nnewlpdead, NULL); + heap_page_prune(relation, buffer, vistest, &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is - * ndeleted minus the number of newly-LP_DEAD-set items. + * presult.ndeleted minus the number of newly-LP_DEAD-set items. * * We derive the number of dead tuples like this to avoid totally * forgetting about items that were set to LP_DEAD, since they @@ -175,9 +173,9 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * tracks ndeleted, since it will set the same LP_DEAD items to * LP_UNUSED separately. */ - if (ndeleted > nnewlpdead) + if (presult.ndeleted > presult.nnewlpdead) pgstat_update_heap_dead_tuples(relation, - ndeleted - nnewlpdead); + presult.ndeleted - presult.nnewlpdead); } /* And release buffer lock */ @@ -204,24 +202,22 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * (see heap_prune_satisfies_vacuum and * HeapTupleSatisfiesVacuum). * - * Sets *nnewlpdead for caller, indicating the number of items that were - * newly set LP_DEAD during prune operation. + * presult contains output parameters needed by callers such as the number of + * tuples removed and the number of line pointers newly marked LP_DEAD. + * heap_page_prune() is responsible for initializing it. * * off_loc is the current offset into the line pointer array while pruning. * This is used by vacuum to populate the error context message. On-access * pruning has no such callback mechanism for populating the error context, so * it passes NULL. When provided by the caller, off_loc is set exclusively by * heap_page_prune(). - * - * Returns the number of tuples deleted from the page during this call. */ -int +void heap_page_prune(Relation relation, Buffer buffer, GlobalVisState *vistest, - int *nnewlpdead, + PruneResult *presult, OffsetNumber *off_loc) { - int ndeleted = 0; Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); OffsetNumber offnum, @@ -247,6 +243,9 @@ heap_page_prune(Relation relation, Buffer buffer, prstate.nredirected = prstate.ndead = prstate.nunused = 0; memset(prstate.marked, 0, sizeof(prstate.marked)); + presult->ndeleted = 0; + presult->nnewlpdead = 0; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -321,7 +320,7 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - ndeleted += heap_prune_chain(buffer, offnum, &prstate); + presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); } /* Clear the offset information once we have processed the given page. */ @@ -422,9 +421,7 @@ heap_page_prune(Relation relation, Buffer buffer, END_CRIT_SECTION(); /* Record number of newly-set-LP_DEAD items for caller */ - *nnewlpdead = prstate.ndead; - - return ndeleted; + presult->nnewlpdead = prstate.ndead; } diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 102cc97358..7ead9cfe9d 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1544,12 +1544,11 @@ lazy_scan_prune(LVRelState *vacrel, ItemId itemid; HeapTupleData tuple; HTSV_Result res; - int tuples_deleted, - tuples_frozen, + PruneResult presult; + int tuples_frozen, lpdead_items, live_tuples, recently_dead_tuples; - int nnewlpdead; HeapPageFreeze pagefrz; int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; @@ -1572,7 +1571,6 @@ retry: pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid; - tuples_deleted = 0; tuples_frozen = 0; lpdead_items = 0; live_tuples = 0; @@ -1581,9 +1579,8 @@ retry: /* * Prune all HOT-update chains in this page. * - * We count tuples removed by the pruning step as tuples_deleted. Its - * final value can be thought of as the number of tuples that have been - * deleted from the table. It should not be confused with lpdead_items; + * We count the number of tuples removed from the page by the pruning step + * in presult.ndeleted. It should not be confused with lpdead_items; * lpdead_items's final value can be thought of as the number of tuples * that were deleted from indexes. * @@ -1591,9 +1588,7 @@ retry: * current offset when populating the error context message, so it is * imperative that we pass its location to heap_page_prune. */ - tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, - &nnewlpdead, - &vacrel->offnum); + heap_page_prune(rel, buf, vacrel->vistest, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1933,7 +1928,7 @@ retry: } /* Finally, add page-local counts to whole-VACUUM counts */ - vacrel->tuples_deleted += tuples_deleted; + vacrel->tuples_deleted += presult.ndeleted; vacrel->tuples_frozen += tuples_frozen; vacrel->lpdead_items += lpdead_items; vacrel->live_tuples += live_tuples; diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 6598c4d7d8..2d3f149e4f 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -191,6 +191,15 @@ typedef struct HeapPageFreeze } HeapPageFreeze; +/* + * Per-page state returned from pruning + */ +typedef struct PruneResult +{ + int ndeleted; /* Number of tuples deleted from the page */ + int nnewlpdead; /* Number of newly LP_DEAD items */ +} PruneResult; + /* ---------------- * function prototypes for heap access method * @@ -284,9 +293,9 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern int heap_page_prune(Relation relation, Buffer buffer, +extern void heap_page_prune(Relation relation, Buffer buffer, struct GlobalVisState *vistest, - int *nnewlpdead, + PruneResult *presult, OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 0656c94416..8c30c0b1b6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2150,6 +2150,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.37.2 [text/x-patch] v4-0003-Reuse-heap_page_prune-tuple-visibility-statuses.patch (10.8K, ../../CAAKRu_YRv4OvT3t4Dw+8UCuji2aVmQxmkJpXvqvTBTWBHEJ-ww@mail.gmail.com/3-v4-0003-Reuse-heap_page_prune-tuple-visibility-statuses.patch) download | inline diff: From 1a621864420cd17db1bef0546990d8298e3a3d34 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Wed, 6 Sep 2023 16:54:41 -0400 Subject: [PATCH v4 3/3] Reuse heap_page_prune() tuple visibility statuses heap_page_prune() obtains the HTSV_Result (tuple visibility status) returned from HeapTupleSatisfiesVacuum() for every tuple on the page and stores them in an array. By making this array available to heap_page_prune()'s caller lazy_scan_prune(), we can avoid an additional call to HeapTupleSatisfiesVacuum() when freezing the tuples and recording LP_DEAD items for vacuum. This saves resources and eliminates the possibility that vacuuming corrupted data results in a hang due to endless retry looping. This replaces the retry mechanism introduced in 8523492d4 to handle cases in which a tuple's inserting transaction aborted between the visibility check in heap_page_prune() and lazy_scan_prune()'s call to HeapTupleSatisfiesVacuum() -- rendering it dead but without a dead line pointer. We can instead reuse the tuple's original visibility status, circumventing any disagreements. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 37 ++++++++++++------------ src/backend/access/heap/vacuumlazy.c | 42 ++++++++-------------------- src/include/access/heapam.h | 11 ++++++++ 3 files changed, 41 insertions(+), 49 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 5ac286e152..256039c6be 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -53,16 +53,6 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; - - /* - * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. - * - * Same indexing as ->marked. - */ - int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneState; /* Local functions */ @@ -71,6 +61,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + int8 *htsv, PruneState *prstate); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, @@ -243,6 +234,10 @@ heap_page_prune(Relation relation, Buffer buffer, prstate.nredirected = prstate.ndead = prstate.nunused = 0; memset(prstate.marked, 0, sizeof(prstate.marked)); + /* + * presult->htsv is not initialized here because all ntuple spots in the + * array will be set either to a valid HTSV_Result value or -1. + */ presult->ndeleted = 0; presult->nnewlpdead = 0; @@ -279,7 +274,7 @@ heap_page_prune(Relation relation, Buffer buffer, /* Nothing to do if slot doesn't contain a tuple */ if (!ItemIdIsNormal(itemid)) { - prstate.htsv[offnum] = -1; + presult->htsv[offnum] = -1; continue; } @@ -295,8 +290,8 @@ heap_page_prune(Relation relation, Buffer buffer, if (off_loc) *off_loc = offnum; - prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, - buffer); + presult->htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, + buffer); } /* Scan the page */ @@ -320,7 +315,8 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); + presult->ndeleted += heap_prune_chain(buffer, offnum, + presult->htsv, &prstate); } /* Clear the offset information once we have processed the given page. */ @@ -449,6 +445,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) /* * Prune specified line pointer or a HOT chain originating at line pointer. * + * Tuple visibility information is provided in htsv. + * * If the item is an index-referenced tuple (i.e. not a heap-only tuple), * the HOT chain is pruned by removing all DEAD tuples at the start of the HOT * chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple. @@ -476,7 +474,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) * Returns the number of tuples (to be) deleted from the page. */ static int -heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) +heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + int8 *htsv, PruneState *prstate) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -497,7 +496,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ if (ItemIdIsNormal(rootlp)) { - Assert(prstate->htsv[rootoffnum] != -1); + Assert(htsv[rootoffnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, rootlp); if (HeapTupleHeaderIsHeapOnly(htup)) @@ -520,7 +519,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) * either here or while following a chain below. Whichever path * gets there first will mark the tuple unused. */ - if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD && + if (htsv[rootoffnum] == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup)) { heap_prune_record_unused(prstate, rootoffnum); @@ -588,7 +587,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) break; Assert(ItemIdIsNormal(lp)); - Assert(prstate->htsv[offnum] != -1); + Assert(htsv[offnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, lp); /* @@ -608,7 +607,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ tupdead = recent_dead = false; - switch ((HTSV_Result) prstate->htsv[offnum]) + switch ((HTSV_Result) htsv[offnum]) { case HEAPTUPLE_DEAD: tupdead = true; diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 7ead9cfe9d..6c328f3ed5 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1524,12 +1524,12 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * of complexity just so we could deal with tuples that were DEAD to VACUUM, * but nevertheless were left with storage after pruning. * - * The approach we take now is to restart pruning when the race condition is - * detected. This allows heap_page_prune() to prune the tuples inserted by - * the now-aborted transaction. This is a little crude, but it guarantees - * that any items that make it into the dead_items array are simple LP_DEAD - * line pointers, and that every remaining item with tuple storage is - * considered as a candidate for freezing. + * As of Postgres 17, we circumvent this problem altogether by reusing the + * result of heap_page_prune()'s visibility check. Without the second call to + * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no + * disagreement. The tuple's TID won't be added to the array of dead tuple TIDs + * for vacuum, thus vacuum will never be tasked with reaping a tuple with + * storage. */ static void lazy_scan_prune(LVRelState *vacrel, @@ -1542,8 +1542,6 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - HeapTupleData tuple; - HTSV_Result res; PruneResult presult; int tuples_frozen, lpdead_items, @@ -1563,8 +1561,6 @@ lazy_scan_prune(LVRelState *vacrel, */ maxoff = PageGetMaxOffsetNumber(page); -retry: - /* Initialize (or reset) page-level state */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; @@ -1604,6 +1600,7 @@ retry: offnum <= maxoff; offnum = OffsetNumberNext(offnum)) { + HeapTupleHeader htup; bool totally_frozen; /* @@ -1646,22 +1643,7 @@ retry: Assert(ItemIdIsNormal(itemid)); - ItemPointerSet(&(tuple.t_self), blkno, offnum); - tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid); - tuple.t_len = ItemIdGetLength(itemid); - tuple.t_tableOid = RelationGetRelid(rel); - - /* - * DEAD tuples are almost always pruned into LP_DEAD line pointers by - * heap_page_prune(), but it's possible that the tuple state changed - * since heap_page_prune() looked. Handle that here by restarting. - * (See comments at the top of function for a full explanation.) - */ - res = HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin, - buf); - - if (unlikely(res == HEAPTUPLE_DEAD)) - goto retry; + htup = (HeapTupleHeader) PageGetItem(page, itemid); /* * The criteria for counting a tuple as live in this block need to @@ -1682,7 +1664,7 @@ retry: * (Cases where we bypass index vacuuming will violate this optimistic * assumption, but the overall impact of that should be negligible.) */ - switch (res) + switch ((HTSV_Result) presult.htsv[offnum]) { case HEAPTUPLE_LIVE: @@ -1704,7 +1686,7 @@ retry: { TransactionId xmin; - if (!HeapTupleHeaderXminCommitted(tuple.t_data)) + if (!HeapTupleHeaderXminCommitted(htup)) { prunestate->all_visible = false; break; @@ -1714,7 +1696,7 @@ retry: * The inserter definitely committed. But is it old enough * that everyone sees it as committed? */ - xmin = HeapTupleHeaderGetXmin(tuple.t_data); + xmin = HeapTupleHeaderGetXmin(htup); if (!TransactionIdPrecedes(xmin, vacrel->cutoffs.OldestXmin)) { @@ -1768,7 +1750,7 @@ retry: prunestate->hastup = true; /* page makes rel truncation unsafe */ /* Tuple with storage -- consider need to freeze */ - if (heap_prepare_freeze_tuple(tuple.t_data, &vacrel->cutoffs, &pagefrz, + if (heap_prepare_freeze_tuple(htup, &vacrel->cutoffs, &pagefrz, &frozen[tuples_frozen], &totally_frozen)) { /* Save prepared freeze plan for later */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2d3f149e4f..864ae561c4 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -198,6 +198,17 @@ typedef struct PruneResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ + + /* + * Tuple visibility is only computed once for each tuple, for correctness + * and efficiency reasons; see comment in heap_page_prune() for details. + * This is of type int8[], instead of HTSV_Result[], so we can use -1 to + * indicate no visibility has been computed, e.g. for LP_DEAD items. + * + * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is + * 1. Otherwise every access would need to subtract 1. + */ + int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneResult; /* ---------------- -- 2.37.2 [text/x-patch] v4-0001-Clarify-usage-of-heap_page_prune-parameter-off_lo.patch (2.8K, ../../CAAKRu_YRv4OvT3t4Dw+8UCuji2aVmQxmkJpXvqvTBTWBHEJ-ww@mail.gmail.com/4-v4-0001-Clarify-usage-of-heap_page_prune-parameter-off_lo.patch) download | inline diff: From 661d9837ba34552fa301dc17b1fd74dd833aec72 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Thu, 7 Sep 2023 17:35:46 -0400 Subject: [PATCH v4 1/3] Clarify usage of heap_page_prune parameter off_loc heap_page_prune() takes an optional parameter, off_loc, where it saves the current offset while pruning. Vacuum passes the address of vacrel->offnum, which is used by vacuum_error_callback() to populate the error context message. On-access pruning has no such callback mechanism, so it passes NULL. When off_loc is not provided the error message context simply omits the offset. Because vacrel->offnum is specifically used by vacuum_error_callback(), it is required that it be passed to heap_page_prune() for the error message to be correctly populated. Clarify this requirement in a comment in lazy_scan_prune() before calling heap_page_prune(). While we are at it, provide some additional detail about off_loc in heap_page_prune()'s function header comment. Discussion: https://postgr.es/m/CA%2BTgmoZ8E7DXzAZkV%3DRF9tfJkQFH%3Da3KkjQhYNY9W26Jh7AN%2Bw%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 7 +++++-- src/backend/access/heap/vacuumlazy.c | 4 ++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 18193efa23..392b54f093 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -207,8 +207,11 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * Sets *nnewlpdead for caller, indicating the number of items that were * newly set LP_DEAD during prune operation. * - * off_loc is the offset location required by the caller to use in error - * callback. + * off_loc is the current offset into the line pointer array while pruning. + * This is used by vacuum to populate the error context message. On-access + * pruning has no such callback mechanism for populating the error context, so + * it passes NULL. When provided by the caller, off_loc is set exclusively by + * heap_page_prune(). * * Returns the number of tuples deleted from the page during this call. */ diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 1a05adfa61..102cc97358 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1586,6 +1586,10 @@ retry: * deleted from the table. It should not be confused with lpdead_items; * lpdead_items's final value can be thought of as the number of tuples * that were deleted from indexes. + * + * vacuum_error_callback() looks specifically at vacrel->offnum for the + * current offset when populating the error context message, so it is + * imperative that we pass its location to heap_page_prune. */ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, &nnewlpdead, -- 2.37.2 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> @ 2023-09-08 15:06 ` Robert Haas <[email protected]> 2023-09-11 12:04 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 1 sibling, 1 reply; 70+ messages in thread From: Robert Haas @ 2023-09-08 15:06 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Thu, Sep 7, 2023 at 6:23 PM Melanie Plageman <[email protected]> wrote: > I mostly wanted to remove the NULL checks because I found them > distracting (so, a stylistic complaint). However, upon further > reflection, I actually think it is better if heap_page_prune_opt() > passes NULL. heap_page_prune() has no error callback mechanism that > could use it, and passing a valid value implies otherwise. Also, as > you said, off_loc will always be InvalidOffsetNumber when > heap_page_prune() returns normally, so having heap_page_prune_opt() > pass a dummy value might actually be more confusing for future > programmers. I'll look at the new patches more next week, but I wanted to comment on this point. I think this is kind of six of one, half a dozen of the other. It's not that hard to spot a variable that's only used in a function call and never initialized beforehand or used afterward, and if someone really feels the need to hammer home the point, they could always name it dummy or dummy_loc or whatever. So this point doesn't really carry a lot of weight with me. I actually think that the proposed change is probably better, but it seems like such a minor improvement that I get slightly reluctant to make it, only because churning the source code for arguable points sometimes annoys other developers. But I also had the thought that maybe it wouldn't be such a terrible idea if heap_page_prune_opt() actually used off_loc for some error reporting goodness. I mean, if HOT pruning fails, and we don't get the detail as to which tuple caused the failure, we can always run VACUUM and it will give us that information, assuming of course that the same failure happens again. But is there any reason why HOT pruning shouldn't include that error detail? If it did, then off_loc would be passed by all callers, at which point we surely would want to get rid of the branches. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-08 15:06 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> @ 2023-09-11 12:04 ` Melanie Plageman <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Melanie Plageman @ 2023-09-11 12:04 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: pgsql-hackers; Andres Freund <[email protected]>; David Geier <[email protected]> On Fri, Sep 8, 2023 at 11:06 AM Robert Haas <[email protected]> wrote: > > On Thu, Sep 7, 2023 at 6:23 PM Melanie Plageman > <[email protected]> wrote: > > I mostly wanted to remove the NULL checks because I found them > > distracting (so, a stylistic complaint). However, upon further > > reflection, I actually think it is better if heap_page_prune_opt() > > passes NULL. heap_page_prune() has no error callback mechanism that > > could use it, and passing a valid value implies otherwise. Also, as > > you said, off_loc will always be InvalidOffsetNumber when > > heap_page_prune() returns normally, so having heap_page_prune_opt() > > pass a dummy value might actually be more confusing for future > > programmers. > > I'll look at the new patches more next week, but I wanted to comment > on this point. I think this is kind of six of one, half a dozen of the > other. It's not that hard to spot a variable that's only used in a > function call and never initialized beforehand or used afterward, and > if someone really feels the need to hammer home the point, they could > always name it dummy or dummy_loc or whatever. So this point doesn't > really carry a lot of weight with me. I actually think that the > proposed change is probably better, but it seems like such a minor > improvement that I get slightly reluctant to make it, only because > churning the source code for arguable points sometimes annoys other > developers. > > But I also had the thought that maybe it wouldn't be such a terrible > idea if heap_page_prune_opt() actually used off_loc for some error > reporting goodness. I mean, if HOT pruning fails, and we don't get the > detail as to which tuple caused the failure, we can always run VACUUM > and it will give us that information, assuming of course that the same > failure happens again. But is there any reason why HOT pruning > shouldn't include that error detail? If it did, then off_loc would be > passed by all callers, at which point we surely would want to get rid > of the branches. This is a good idea. I will work on a separate patch set to add an error context callback for on-access HOT pruning. - Melanie ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> @ 2023-09-13 17:29 ` Andres Freund <[email protected]> 2023-09-21 19:53 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 1 sibling, 1 reply; 70+ messages in thread From: Andres Freund @ 2023-09-13 17:29 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers; David Geier <[email protected]> Hi, On 2023-09-07 18:23:22 -0400, Melanie Plageman wrote: > From e986940e546171d1f1d06f62a101d695a8481e7a Mon Sep 17 00:00:00 2001 > From: Melanie Plageman <[email protected]> > Date: Wed, 6 Sep 2023 14:57:20 -0400 > Subject: [PATCH v4 2/3] Move heap_page_prune output parameters into struct > > Add PruneResult, a structure containing the output parameters and result > of heap_page_prune(). Reorganizing the results of heap_page_prune() into > a struct simplifies the function signature and provides a location for > future commits to store additional output parameters. > > Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com > --- > src/backend/access/heap/pruneheap.c | 33 +++++++++++++--------------- > src/backend/access/heap/vacuumlazy.c | 17 +++++--------- > src/include/access/heapam.h | 13 +++++++++-- > src/tools/pgindent/typedefs.list | 1 + > 4 files changed, 33 insertions(+), 31 deletions(-) > > diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c > index 392b54f093..5ac286e152 100644 > --- a/src/backend/access/heap/pruneheap.c > +++ b/src/backend/access/heap/pruneheap.c > @@ -155,15 +155,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) > */ > if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) > { > - int ndeleted, > - nnewlpdead; > + PruneResult presult; > > - ndeleted = heap_page_prune(relation, buffer, vistest, > - &nnewlpdead, NULL); > + heap_page_prune(relation, buffer, vistest, &presult, NULL); > > /* > * Report the number of tuples reclaimed to pgstats. This is > - * ndeleted minus the number of newly-LP_DEAD-set items. > + * presult.ndeleted minus the number of newly-LP_DEAD-set items. > * > * We derive the number of dead tuples like this to avoid totally > * forgetting about items that were set to LP_DEAD, since they > @@ -175,9 +173,9 @@ heap_page_prune_opt(Relation relation, Buffer buffer) > * tracks ndeleted, since it will set the same LP_DEAD items to > * LP_UNUSED separately. > */ > - if (ndeleted > nnewlpdead) > + if (presult.ndeleted > presult.nnewlpdead) > pgstat_update_heap_dead_tuples(relation, > - ndeleted - nnewlpdead); > + presult.ndeleted - presult.nnewlpdead); > } > > /* And release buffer lock */ > @@ -204,24 +202,22 @@ heap_page_prune_opt(Relation relation, Buffer buffer) > * (see heap_prune_satisfies_vacuum and > * HeapTupleSatisfiesVacuum). > * > - * Sets *nnewlpdead for caller, indicating the number of items that were > - * newly set LP_DEAD during prune operation. > + * presult contains output parameters needed by callers such as the number of > + * tuples removed and the number of line pointers newly marked LP_DEAD. > + * heap_page_prune() is responsible for initializing it. > * > * off_loc is the current offset into the line pointer array while pruning. > * This is used by vacuum to populate the error context message. On-access > * pruning has no such callback mechanism for populating the error context, so > * it passes NULL. When provided by the caller, off_loc is set exclusively by > * heap_page_prune(). > - * > - * Returns the number of tuples deleted from the page during this call. > */ > -int > +void > heap_page_prune(Relation relation, Buffer buffer, > GlobalVisState *vistest, > - int *nnewlpdead, > + PruneResult *presult, > OffsetNumber *off_loc) > { > - int ndeleted = 0; > Page page = BufferGetPage(buffer); > BlockNumber blockno = BufferGetBlockNumber(buffer); > OffsetNumber offnum, > @@ -247,6 +243,9 @@ heap_page_prune(Relation relation, Buffer buffer, > prstate.nredirected = prstate.ndead = prstate.nunused = 0; > memset(prstate.marked, 0, sizeof(prstate.marked)); > > + presult->ndeleted = 0; > + presult->nnewlpdead = 0; > + > maxoff = PageGetMaxOffsetNumber(page); > tup.t_tableOid = RelationGetRelid(prstate.rel); > > @@ -321,7 +320,7 @@ heap_page_prune(Relation relation, Buffer buffer, > continue; > > /* Process this item or chain of items */ > - ndeleted += heap_prune_chain(buffer, offnum, &prstate); > + presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); > } > > /* Clear the offset information once we have processed the given page. */ > @@ -422,9 +421,7 @@ heap_page_prune(Relation relation, Buffer buffer, > END_CRIT_SECTION(); > > /* Record number of newly-set-LP_DEAD items for caller */ > - *nnewlpdead = prstate.ndead; > - > - return ndeleted; > + presult->nnewlpdead = prstate.ndead; > } > > > diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c > index 102cc97358..7ead9cfe9d 100644 > --- a/src/backend/access/heap/vacuumlazy.c > +++ b/src/backend/access/heap/vacuumlazy.c > @@ -1544,12 +1544,11 @@ lazy_scan_prune(LVRelState *vacrel, > ItemId itemid; > HeapTupleData tuple; > HTSV_Result res; > - int tuples_deleted, > - tuples_frozen, > + PruneResult presult; > + int tuples_frozen, > lpdead_items, > live_tuples, > recently_dead_tuples; > - int nnewlpdead; > HeapPageFreeze pagefrz; > int64 fpi_before = pgWalUsage.wal_fpi; > OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; > @@ -1572,7 +1571,6 @@ retry: > pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; > pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid; > pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid; > - tuples_deleted = 0; > tuples_frozen = 0; > lpdead_items = 0; > live_tuples = 0; > @@ -1581,9 +1579,8 @@ retry: > /* > * Prune all HOT-update chains in this page. > * > - * We count tuples removed by the pruning step as tuples_deleted. Its > - * final value can be thought of as the number of tuples that have been > - * deleted from the table. It should not be confused with lpdead_items; > + * We count the number of tuples removed from the page by the pruning step > + * in presult.ndeleted. It should not be confused with lpdead_items; > * lpdead_items's final value can be thought of as the number of tuples > * that were deleted from indexes. > * > @@ -1591,9 +1588,7 @@ retry: > * current offset when populating the error context message, so it is > * imperative that we pass its location to heap_page_prune. > */ > - tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, > - &nnewlpdead, > - &vacrel->offnum); > + heap_page_prune(rel, buf, vacrel->vistest, &presult, &vacrel->offnum); > > /* > * Now scan the page to collect LP_DEAD items and check for tuples > @@ -1933,7 +1928,7 @@ retry: > } > > /* Finally, add page-local counts to whole-VACUUM counts */ > - vacrel->tuples_deleted += tuples_deleted; > + vacrel->tuples_deleted += presult.ndeleted; > vacrel->tuples_frozen += tuples_frozen; > vacrel->lpdead_items += lpdead_items; > vacrel->live_tuples += live_tuples; > diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h > index 6598c4d7d8..2d3f149e4f 100644 > --- a/src/include/access/heapam.h > +++ b/src/include/access/heapam.h > @@ -191,6 +191,15 @@ typedef struct HeapPageFreeze > > } HeapPageFreeze; > > +/* > + * Per-page state returned from pruning > + */ > +typedef struct PruneResult > +{ > + int ndeleted; /* Number of tuples deleted from the page */ > + int nnewlpdead; /* Number of newly LP_DEAD items */ > +} PruneResult; I think it might be worth making the names a bit less ambiguous than they were. It's a bit odd that one has "new" in the name, the other doesn't, despite both being about newly marked things. And "deleted" seems somewhat ambiguous, it could also be understood as marking things LP_DEAD. Maybe nnewunused? > static int heap_prune_chain(Buffer buffer, > OffsetNumber rootoffnum, > + int8 *htsv, > PruneState *prstate); Hm, do we really want to pass this explicitly to a bunch of functions? Seems like it might be better to either pass the PruneResult around or to have a pointer in PruneState? > /* > * The criteria for counting a tuple as live in this block need to > @@ -1682,7 +1664,7 @@ retry: > * (Cases where we bypass index vacuuming will violate this optimistic > * assumption, but the overall impact of that should be negligible.) > */ > - switch (res) > + switch ((HTSV_Result) presult.htsv[offnum]) > { > case HEAPTUPLE_LIVE: I think we should assert that we have a valid HTSV_Result here, i.e. not -1. You could wrap the cast and Assert into an inline funciton as well. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-13 17:29 ` Re: Eliminate redundant tuple visibility check in vacuum Andres Freund <[email protected]> @ 2023-09-21 19:53 ` Robert Haas <[email protected]> 2023-09-21 20:07 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-28 12:46 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 0 siblings, 2 replies; 70+ messages in thread From: Robert Haas @ 2023-09-21 19:53 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Melanie Plageman <[email protected]>; pgsql-hackers; David Geier <[email protected]> [ sorry for the delay getting back to this ] On Wed, Sep 13, 2023 at 1:29 PM Andres Freund <[email protected]> wrote: > I think it might be worth making the names a bit less ambiguous than they > were. It's a bit odd that one has "new" in the name, the other doesn't, > despite both being about newly marked things. And "deleted" seems somewhat > ambiguous, it could also be understood as marking things LP_DEAD. Maybe > nnewunused? I like it the better the way Melanie did it. The current name may not be for the best, but that can be changed some other time, in a separate patch, if someone likes. For now, changing the name seems like a can of worms we don't need to open; the existing names have precedent on their side if nothing else. > > static int heap_prune_chain(Buffer buffer, > > OffsetNumber rootoffnum, > > + int8 *htsv, > > PruneState *prstate); > > Hm, do we really want to pass this explicitly to a bunch of functions? Seems > like it might be better to either pass the PruneResult around or to have a > pointer in PruneState? As far as I can see, 0002 adds it to one function (heap_page_pune) and 0003 adds it to one more (heap_prune_chain). That's not much of a bunch. > > /* > > * The criteria for counting a tuple as live in this block need to > > @@ -1682,7 +1664,7 @@ retry: > > * (Cases where we bypass index vacuuming will violate this optimistic > > * assumption, but the overall impact of that should be negligible.) > > */ > > - switch (res) > > + switch ((HTSV_Result) presult.htsv[offnum]) > > { > > case HEAPTUPLE_LIVE: > > I think we should assert that we have a valid HTSV_Result here, i.e. not > -1. You could wrap the cast and Assert into an inline funciton as well. This isn't a bad idea, although I don't find it completely necessary either. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-13 17:29 ` Re: Eliminate redundant tuple visibility check in vacuum Andres Freund <[email protected]> 2023-09-21 19:53 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> @ 2023-09-21 20:07 ` Robert Haas <[email protected]> 1 sibling, 0 replies; 70+ messages in thread From: Robert Haas @ 2023-09-21 20:07 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Melanie Plageman <[email protected]>; pgsql-hackers; David Geier <[email protected]> On Thu, Sep 21, 2023 at 3:53 PM Robert Haas <[email protected]> wrote: > > > static int heap_prune_chain(Buffer buffer, > > > OffsetNumber rootoffnum, > > > + int8 *htsv, > > > PruneState *prstate); > > > > Hm, do we really want to pass this explicitly to a bunch of functions? Seems > > like it might be better to either pass the PruneResult around or to have a > > pointer in PruneState? > > As far as I can see, 0002 adds it to one function (heap_page_pune) and > 0003 adds it to one more (heap_prune_chain). That's not much of a > bunch. I didn't read this carefully enough. Actually, heap_prune_chain() is the *only* function that gets int8 *htsv as an argument. I don't understand how that's a bunch ... unless there are later patches not shown here that you're worried abot. What happens in 0002 is a function getting PruneResult * as an argument, not int8 *htsv. Honestly I think 0002 and 0003 are ready to commit, if you're not too opposed to them, or if we can find some relatively small changes that would address your objections. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-13 17:29 ` Re: Eliminate redundant tuple visibility check in vacuum Andres Freund <[email protected]> 2023-09-21 19:53 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> @ 2023-09-28 12:46 ` Melanie Plageman <[email protected]> 2023-09-28 15:25 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 1 sibling, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2023-09-28 12:46 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers; David Geier <[email protected]> On Thu, Sep 21, 2023 at 3:53 PM Robert Haas <[email protected]> wrote: > > On Wed, Sep 13, 2023 at 1:29 PM Andres Freund <[email protected]> wrote: > > > > /* > > > * The criteria for counting a tuple as live in this block need to > > > @@ -1682,7 +1664,7 @@ retry: > > > * (Cases where we bypass index vacuuming will violate this optimistic > > > * assumption, but the overall impact of that should be negligible.) > > > */ > > > - switch (res) > > > + switch ((HTSV_Result) presult.htsv[offnum]) > > > { > > > case HEAPTUPLE_LIVE: > > > > I think we should assert that we have a valid HTSV_Result here, i.e. not > > -1. You could wrap the cast and Assert into an inline funciton as well. > > This isn't a bad idea, although I don't find it completely necessary either. Attached v5 does this. Even though a value of -1 would hit the default switch case and error out, I can see the argument for this validation -- as all other places switching on an HTSV_Result are doing so on a value which was always an HTSV_Result. Once I started writing the function comment, however, I felt a bit awkward. In order to make the function available to both pruneheap.c and vacuumlazy.c, I had to put it in a header file. Writing a function, available to anyone including heapam.h, which takes an int and returns an HTSV_Result feels a bit odd. Do we want it to be common practice to use an int value outside the valid enum range to store "status not yet computed" for HTSV_Results? Anyway, on a tactical note, I added the inline function to heapam.h below the PruneResult definition since it is fairly tightly coupled to the htsv array in PruneResult. All of the function prototypes are under a comment that says "function prototypes for heap access method" -- which didn't feel like an accurate description of this function. I wonder if it makes sense to have pruneheap.c include vacuum.h and move pruning specific stuff like this helper and PruneResult over there? I can't remember why I didn't do this before, but maybe there is a reason not to? I also wasn't sure if I needed to forward declare the inline function or not. Oh, and, one more note. I've dropped the former patch 0001 which changed the function comment about off_loc above heap_page_prune(). I have plans to write a separate patch adding an error context callback for HOT pruning with the offset number and would include such a change in that patch. - Melanie Attachments: [text/x-patch] v5-0002-Reuse-heap_page_prune-tuple-visibility-statuses.patch (11.3K, ../../CAAKRu_aUQcRvrt=ka4qCMJVRjGUvLCJjiiu_6iRQAq2mb9LOHA@mail.gmail.com/2-v5-0002-Reuse-heap_page_prune-tuple-visibility-statuses.patch) download | inline diff: From aa1ec4fa1a689f9c6328d02fd13af1a42ecb0396 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Wed, 6 Sep 2023 16:54:41 -0400 Subject: [PATCH v5 2/2] Reuse heap_page_prune() tuple visibility statuses heap_page_prune() obtains the HTSV_Result (tuple visibility status) returned from HeapTupleSatisfiesVacuum() for every tuple on the page and stores them in an array. By making this array available to heap_page_prune()'s caller lazy_scan_prune(), we can avoid an additional call to HeapTupleSatisfiesVacuum() when freezing the tuples and recording LP_DEAD items for vacuum. This saves resources and eliminates the possibility that vacuuming corrupted data results in a hang due to endless retry looping. This replaces the retry mechanism introduced in 8523492d4 to handle cases in which a tuple's inserting transaction aborted between the visibility check in heap_page_prune() and lazy_scan_prune()'s call to HeapTupleSatisfiesVacuum() -- rendering it dead but without a dead line pointer. We can instead reuse the tuple's original visibility status, circumventing any disagreements. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 36 +++++++++++------------- src/backend/access/heap/vacuumlazy.c | 42 ++++++++-------------------- src/include/access/heapam.h | 25 +++++++++++++++++ 3 files changed, 54 insertions(+), 49 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index d5892a2db4..c5f1abd95a 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -53,16 +53,6 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; - - /* - * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. - * - * Same indexing as ->marked. - */ - int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneState; /* Local functions */ @@ -71,6 +61,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + int8 *htsv, PruneState *prstate); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, @@ -240,6 +231,10 @@ heap_page_prune(Relation relation, Buffer buffer, prstate.nredirected = prstate.ndead = prstate.nunused = 0; memset(prstate.marked, 0, sizeof(prstate.marked)); + /* + * presult->htsv is not initialized here because all ntuple spots in the + * array will be set either to a valid HTSV_Result value or -1. + */ presult->ndeleted = 0; presult->nnewlpdead = 0; @@ -276,7 +271,7 @@ heap_page_prune(Relation relation, Buffer buffer, /* Nothing to do if slot doesn't contain a tuple */ if (!ItemIdIsNormal(itemid)) { - prstate.htsv[offnum] = -1; + presult->htsv[offnum] = -1; continue; } @@ -292,8 +287,8 @@ heap_page_prune(Relation relation, Buffer buffer, if (off_loc) *off_loc = offnum; - prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, - buffer); + presult->htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup, + buffer); } /* Scan the page */ @@ -317,7 +312,8 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); + presult->ndeleted += heap_prune_chain(buffer, offnum, + presult->htsv, &prstate); } /* Clear the offset information once we have processed the given page. */ @@ -446,6 +442,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) /* * Prune specified line pointer or a HOT chain originating at line pointer. * + * Tuple visibility information is provided in htsv. + * * If the item is an index-referenced tuple (i.e. not a heap-only tuple), * the HOT chain is pruned by removing all DEAD tuples at the start of the HOT * chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple. @@ -473,7 +471,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) * Returns the number of tuples (to be) deleted from the page. */ static int -heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) +heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, + int8 *htsv, PruneState *prstate) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -494,7 +493,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ if (ItemIdIsNormal(rootlp)) { - Assert(prstate->htsv[rootoffnum] != -1); + Assert(htsv[rootoffnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, rootlp); if (HeapTupleHeaderIsHeapOnly(htup)) @@ -517,7 +516,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) * either here or while following a chain below. Whichever path * gets there first will mark the tuple unused. */ - if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD && + if (htsv[rootoffnum] == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup)) { heap_prune_record_unused(prstate, rootoffnum); @@ -585,7 +584,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) break; Assert(ItemIdIsNormal(lp)); - Assert(prstate->htsv[offnum] != -1); htup = (HeapTupleHeader) PageGetItem(dp, lp); /* @@ -605,7 +603,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate) */ tupdead = recent_dead = false; - switch ((HTSV_Result) prstate->htsv[offnum]) + switch (htsv_get_valid_status(htsv[offnum])) { case HEAPTUPLE_DEAD: tupdead = true; diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index fa77ef7f4a..b8268a5cef 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1524,12 +1524,12 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * of complexity just so we could deal with tuples that were DEAD to VACUUM, * but nevertheless were left with storage after pruning. * - * The approach we take now is to restart pruning when the race condition is - * detected. This allows heap_page_prune() to prune the tuples inserted by - * the now-aborted transaction. This is a little crude, but it guarantees - * that any items that make it into the dead_items array are simple LP_DEAD - * line pointers, and that every remaining item with tuple storage is - * considered as a candidate for freezing. + * As of Postgres 17, we circumvent this problem altogether by reusing the + * result of heap_page_prune()'s visibility check. Without the second call to + * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no + * disagreement. The tuple's TID won't be added to the array of dead tuple TIDs + * for vacuum, thus vacuum will never be tasked with reaping a tuple with + * storage. */ static void lazy_scan_prune(LVRelState *vacrel, @@ -1542,8 +1542,6 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - HeapTupleData tuple; - HTSV_Result res; PruneResult presult; int tuples_frozen, lpdead_items, @@ -1563,8 +1561,6 @@ lazy_scan_prune(LVRelState *vacrel, */ maxoff = PageGetMaxOffsetNumber(page); -retry: - /* Initialize (or reset) page-level state */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; @@ -1600,6 +1596,7 @@ retry: offnum <= maxoff; offnum = OffsetNumberNext(offnum)) { + HeapTupleHeader htup; bool totally_frozen; /* @@ -1642,22 +1639,7 @@ retry: Assert(ItemIdIsNormal(itemid)); - ItemPointerSet(&(tuple.t_self), blkno, offnum); - tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid); - tuple.t_len = ItemIdGetLength(itemid); - tuple.t_tableOid = RelationGetRelid(rel); - - /* - * DEAD tuples are almost always pruned into LP_DEAD line pointers by - * heap_page_prune(), but it's possible that the tuple state changed - * since heap_page_prune() looked. Handle that here by restarting. - * (See comments at the top of function for a full explanation.) - */ - res = HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin, - buf); - - if (unlikely(res == HEAPTUPLE_DEAD)) - goto retry; + htup = (HeapTupleHeader) PageGetItem(page, itemid); /* * The criteria for counting a tuple as live in this block need to @@ -1678,7 +1660,7 @@ retry: * (Cases where we bypass index vacuuming will violate this optimistic * assumption, but the overall impact of that should be negligible.) */ - switch (res) + switch (htsv_get_valid_status(presult.htsv[offnum])) { case HEAPTUPLE_LIVE: @@ -1700,7 +1682,7 @@ retry: { TransactionId xmin; - if (!HeapTupleHeaderXminCommitted(tuple.t_data)) + if (!HeapTupleHeaderXminCommitted(htup)) { prunestate->all_visible = false; break; @@ -1710,7 +1692,7 @@ retry: * The inserter definitely committed. But is it old enough * that everyone sees it as committed? */ - xmin = HeapTupleHeaderGetXmin(tuple.t_data); + xmin = HeapTupleHeaderGetXmin(htup); if (!TransactionIdPrecedes(xmin, vacrel->cutoffs.OldestXmin)) { @@ -1764,7 +1746,7 @@ retry: prunestate->hastup = true; /* page makes rel truncation unsafe */ /* Tuple with storage -- consider need to freeze */ - if (heap_prepare_freeze_tuple(tuple.t_data, &vacrel->cutoffs, &pagefrz, + if (heap_prepare_freeze_tuple(htup, &vacrel->cutoffs, &pagefrz, &frozen[tuples_frozen], &totally_frozen)) { /* Save prepared freeze plan for later */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2d3f149e4f..62fac1d5d2 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -198,8 +198,33 @@ typedef struct PruneResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ + + /* + * Tuple visibility is only computed once for each tuple, for correctness + * and efficiency reasons; see comment in heap_page_prune() for details. + * This is of type int8[], instead of HTSV_Result[], so we can use -1 to + * indicate no visibility has been computed, e.g. for LP_DEAD items. + * + * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is + * 1. Otherwise every access would need to subtract 1. + */ + int8 htsv[MaxHeapTuplesPerPage + 1]; } PruneResult; +/* + * Pruning calculates tuple visibility once and saves the results in an array + * of int8. See PruneResult.htsv for details. This helper function is meant to + * guard against examining visibility status array members which have not yet + * been computed. + */ +static inline HTSV_Result +htsv_get_valid_status(int status) +{ + Assert(status >= HEAPTUPLE_DEAD && + status <= HEAPTUPLE_DELETE_IN_PROGRESS); + return (HTSV_Result) status; +} + /* ---------------- * function prototypes for heap access method * -- 2.37.2 [text/x-patch] v5-0001-Move-heap_page_prune-output-parameters-into-struc.patch (7.5K, ../../CAAKRu_aUQcRvrt=ka4qCMJVRjGUvLCJjiiu_6iRQAq2mb9LOHA@mail.gmail.com/3-v5-0001-Move-heap_page_prune-output-parameters-into-struc.patch) download | inline diff: From 430ff3458f0ac523f1c623d9bc2669981cc0e536 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Wed, 6 Sep 2023 14:57:20 -0400 Subject: [PATCH v5 1/2] Move heap_page_prune output parameters into struct Add PruneResult, a structure containing the output parameters and result of heap_page_prune(). Reorganizing the results of heap_page_prune() into a struct simplifies the function signature and provides a location for future commits to store additional output parameters. Discussion: https://postgr.es/m/CAAKRu_br124qsGJieuYA0nGjywEukhK1dKBfRdby_4yY3E9SXA%40mail.gmail.com --- src/backend/access/heap/pruneheap.c | 33 +++++++++++++--------------- src/backend/access/heap/vacuumlazy.c | 17 +++++--------- src/include/access/heapam.h | 13 +++++++++-- src/tools/pgindent/typedefs.list | 1 + 4 files changed, 33 insertions(+), 31 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 18193efa23..d5892a2db4 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -155,15 +155,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - int ndeleted, - nnewlpdead; + PruneResult presult; - ndeleted = heap_page_prune(relation, buffer, vistest, - &nnewlpdead, NULL); + heap_page_prune(relation, buffer, vistest, &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is - * ndeleted minus the number of newly-LP_DEAD-set items. + * presult.ndeleted minus the number of newly-LP_DEAD-set items. * * We derive the number of dead tuples like this to avoid totally * forgetting about items that were set to LP_DEAD, since they @@ -175,9 +173,9 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * tracks ndeleted, since it will set the same LP_DEAD items to * LP_UNUSED separately. */ - if (ndeleted > nnewlpdead) + if (presult.ndeleted > presult.nnewlpdead) pgstat_update_heap_dead_tuples(relation, - ndeleted - nnewlpdead); + presult.ndeleted - presult.nnewlpdead); } /* And release buffer lock */ @@ -204,21 +202,19 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * (see heap_prune_satisfies_vacuum and * HeapTupleSatisfiesVacuum). * - * Sets *nnewlpdead for caller, indicating the number of items that were - * newly set LP_DEAD during prune operation. - * * off_loc is the offset location required by the caller to use in error * callback. * - * Returns the number of tuples deleted from the page during this call. + * presult contains output parameters needed by callers such as the number of + * tuples removed and the number of line pointers newly marked LP_DEAD. + * heap_page_prune() is responsible for initializing it. */ -int +void heap_page_prune(Relation relation, Buffer buffer, GlobalVisState *vistest, - int *nnewlpdead, + PruneResult *presult, OffsetNumber *off_loc) { - int ndeleted = 0; Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); OffsetNumber offnum, @@ -244,6 +240,9 @@ heap_page_prune(Relation relation, Buffer buffer, prstate.nredirected = prstate.ndead = prstate.nunused = 0; memset(prstate.marked, 0, sizeof(prstate.marked)); + presult->ndeleted = 0; + presult->nnewlpdead = 0; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -318,7 +317,7 @@ heap_page_prune(Relation relation, Buffer buffer, continue; /* Process this item or chain of items */ - ndeleted += heap_prune_chain(buffer, offnum, &prstate); + presult->ndeleted += heap_prune_chain(buffer, offnum, &prstate); } /* Clear the offset information once we have processed the given page. */ @@ -419,9 +418,7 @@ heap_page_prune(Relation relation, Buffer buffer, END_CRIT_SECTION(); /* Record number of newly-set-LP_DEAD items for caller */ - *nnewlpdead = prstate.ndead; - - return ndeleted; + presult->nnewlpdead = prstate.ndead; } diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index b3c24d68f0..fa77ef7f4a 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -1544,12 +1544,11 @@ lazy_scan_prune(LVRelState *vacrel, ItemId itemid; HeapTupleData tuple; HTSV_Result res; - int tuples_deleted, - tuples_frozen, + PruneResult presult; + int tuples_frozen, lpdead_items, live_tuples, recently_dead_tuples; - int nnewlpdead; HeapPageFreeze pagefrz; int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; @@ -1572,7 +1571,6 @@ retry: pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; pagefrz.NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid; - tuples_deleted = 0; tuples_frozen = 0; lpdead_items = 0; live_tuples = 0; @@ -1581,15 +1579,12 @@ retry: /* * Prune all HOT-update chains in this page. * - * We count tuples removed by the pruning step as tuples_deleted. Its - * final value can be thought of as the number of tuples that have been - * deleted from the table. It should not be confused with lpdead_items; + * We count the number of tuples removed from the page by the pruning step + * in presult.ndeleted. It should not be confused with lpdead_items; * lpdead_items's final value can be thought of as the number of tuples * that were deleted from indexes. */ - tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, - &nnewlpdead, - &vacrel->offnum); + heap_page_prune(rel, buf, vacrel->vistest, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1929,7 +1924,7 @@ retry: } /* Finally, add page-local counts to whole-VACUUM counts */ - vacrel->tuples_deleted += tuples_deleted; + vacrel->tuples_deleted += presult.ndeleted; vacrel->tuples_frozen += tuples_frozen; vacrel->lpdead_items += lpdead_items; vacrel->live_tuples += live_tuples; diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 6598c4d7d8..2d3f149e4f 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -191,6 +191,15 @@ typedef struct HeapPageFreeze } HeapPageFreeze; +/* + * Per-page state returned from pruning + */ +typedef struct PruneResult +{ + int ndeleted; /* Number of tuples deleted from the page */ + int nnewlpdead; /* Number of newly LP_DEAD items */ +} PruneResult; + /* ---------------- * function prototypes for heap access method * @@ -284,9 +293,9 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern int heap_page_prune(Relation relation, Buffer buffer, +extern void heap_page_prune(Relation relation, Buffer buffer, struct GlobalVisState *vistest, - int *nnewlpdead, + PruneResult *presult, OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index b5bbdd1608..8de90c4958 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2151,6 +2151,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.37.2 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Eliminate redundant tuple visibility check in vacuum 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-06 21:21 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 19:09 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-07 22:23 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-13 17:29 ` Re: Eliminate redundant tuple visibility check in vacuum Andres Freund <[email protected]> 2023-09-21 19:53 ` Re: Eliminate redundant tuple visibility check in vacuum Robert Haas <[email protected]> 2023-09-28 12:46 ` Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> @ 2023-09-28 15:25 ` Robert Haas <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Robert Haas @ 2023-09-28 15:25 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers; David Geier <[email protected]> On Thu, Sep 28, 2023 at 8:46 AM Melanie Plageman <[email protected]> wrote: > Once I started writing the function comment, however, I felt a bit > awkward. In order to make the function available to both pruneheap.c > and vacuumlazy.c, I had to put it in a header file. Writing a > function, available to anyone including heapam.h, which takes an int > and returns an HTSV_Result feels a bit odd. Do we want it to be common > practice to use an int value outside the valid enum range to store > "status not yet computed" for HTSV_Results? I noticed the awkwardness of that return convention when I reviewed the first version of this patch, but I decided it wasn't worth spending time discussing. To avoid it, we'd either need to add a new HTSV_Result that is only used here, or add a new type HTSV_Result_With_An_Extra_Value and translate between the two, or pass back a boolean + an enum instead of an array of int8. And all of those seem to me to suck -- the first two are messy and the third would make the return value much wider. So, no, I don't really like this, but also, what would actually be any better? Also, IMV at least, it's more of an issue of it being sort of ugly than of anything becoming common practice, because how many callers of heap_page_prune() are there ever going to be? AFAIK, we've only ever had two since forever, and even if we grow one or two more at some point, that's still not that many. I went ahead and committed 0001. If Andres still wants to push for more renaming there, that can be a follow-up patch. And we can see if he or anyone else has any comments on this new version of 0002. To me we're down into the level of details that probably don't matter very much one way or the other, but others may disagree. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin @ 2025-06-12 08:10 John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: John Naylor @ 2025-06-12 08:10 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Thu, Aug 1, 2024 at 4:47 AM Melanie Plageman <[email protected]> wrote: > > On Wed, Jul 31, 2024 at 4:38 PM Peter Geoghegan <[email protected]> wrote: > > > > On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman > > <[email protected]> wrote: > > > In back branches starting with 14, failing to remove tuples older than > > > OldestXmin during pruning caused vacuum to infinitely loop in > > > lazy_scan_prune(), as investigated on this [1] thread. > > > > Shouldn't somebody remove the entry that we have for this issue under > > https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items#Older_bugs_affecting_stable_branches? > > Thanks for the reminder. Done! Hi, are we still lacking test coverage for this on v17 and up? -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> @ 2025-06-12 22:23 ` Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2025-06-12 22:23 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Thu, Jun 12, 2025 at 4:11 AM John Naylor <[email protected]> wrote: > > Hi, are we still lacking test coverage for this on v17 and up? Yep. I have been procrastinating on swapping all this back into my brain. I'll start working on it today. I think I just take the reverted test and make the UPDATE a DELETE, set mwm 64 kB, make the table have 9000 rows and maybe consider using your: delete from test where ctid::text like '%,2__)'; But, unfortunately, to be able to commit it, I'll have to remember everything about this. I'll begin now... - Melanie ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> @ 2025-06-13 14:12 ` John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: John Naylor @ 2025-06-13 14:12 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Fri, Jun 13, 2025 at 5:23 AM Melanie Plageman <[email protected]> wrote: > Yep. I have been procrastinating on swapping all this back into my > brain. I'll start working on it today. I think I just take the > reverted test and make the UPDATE a DELETE, set mwm 64 kB, make the > table have 9000 rows and maybe consider using your: > > delete from test where ctid::text like '%,2__)'; (Swapping this part back in my brain as well...) I actually don't think we need that where clause anymore since mwm can be super low now, and it's a bit mysterious what it was trying to accomplish. Maybe we can just use the lowest fill factor to reduce WAL -- having a few dozen pages should push it over the memory limit, regardless of how many dead tuples are on each pages. -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> @ 2025-06-16 14:58 ` Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2025-06-16 14:58 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Fri, Jun 13, 2025 at 10:12 AM John Naylor <[email protected]> wrote: > > (Swapping this part back in my brain as well...) I actually don't > think we need that where clause anymore since mwm can be super low > now, and it's a bit mysterious what it was trying to accomplish. Maybe > we can just use the lowest fill factor to reduce WAL -- having a few > dozen pages should push it over the memory limit, regardless of how > many dead tuples are on each pages. Test in attached patch seems to do the job on 32 bit and 64 bit when tested. - Melanie Attachments: [text/x-patch] v1-Test-that-vacuum-removes-tuples-older-than-OldestXmi.patch (12.8K, ../../CAAKRu_ZJBkidusDut6i=bDCiXzJEp93GC1+NFaZt4eqanYF3Kw@mail.gmail.com/2-v1-Test-that-vacuum-removes-tuples-older-than-OldestXmi.patch) download | inline diff: From 17485392a10d12f1c04d8dddf90c1c89f641ee95 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Fri, 13 Jun 2025 09:25:40 -0400 Subject: [PATCH] Test that vacuum removes tuples older than OldestXmin If vacuum fails to prune a tuple killed before OldestXmin, it will decide to freeze its xmax and later error out in pre-freeze checks. Add a test reproducing this scenario to the recovery suite which creates a table on a primary, updates the table to generate dead tuples for vacuum, and then, during the vacuum, uses a replica to force GlobalVisState->maybe_needed on the primary to move backwards and precede the value of OldestXmin set at the beginning of vacuuming the table. This test is coverage for a case fixed in 83c39a1f7f3. The test was originally committed to master in aa607980aee but later reverted in efcbb76efe4 due to test instability. The test requires multiple index passes. In Postgres 17+, vacuum uses a TID store for the dead TIDs that is very space efficient. With the old minimum maintenance_work_mem of 1 MB, it required a large number of dead rows to generate enough dead TIDs to force multiple index vacuuming passes. Once the source code changes were made to allow a minimum maintenance_work_mem value of 64kB, the test could be made much faster and more stable. Author: Melanie Plageman Reviewed-by: Peter Geoghegan DiscussioN: https://postgr.es/m/CAAKRu_ZhG5NEQU-h7m%3DaeocxRze4ALt5swuKM45bN0HRQBccew%40mail.gmail.com --- src/test/recovery/meson.build | 3 +- .../recovery/t/048_vacuum_horizon_floor.pl | 266 ++++++++++++++++++ 2 files changed, 268 insertions(+), 1 deletion(-) create mode 100644 src/test/recovery/t/048_vacuum_horizon_floor.pl diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 92429d28402..52993c32dbb 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -55,7 +55,8 @@ tests += { 't/044_invalidate_inactive_slots.pl', 't/045_archive_restartpoint.pl', 't/046_checkpoint_logical_slot.pl', - 't/047_checkpoint_physical_slot.pl' + 't/047_checkpoint_physical_slot.pl', + 't/048_vacuum_horizon_floor.pl' ], }, } diff --git a/src/test/recovery/t/048_vacuum_horizon_floor.pl b/src/test/recovery/t/048_vacuum_horizon_floor.pl new file mode 100644 index 00000000000..141db78f87c --- /dev/null +++ b/src/test/recovery/t/048_vacuum_horizon_floor.pl @@ -0,0 +1,266 @@ +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use Test::More; + +# Test that vacuum prunes away all dead tuples killed before OldestXmin +# +# This test creates a table on a primary, updates the table to generate dead +# tuples for vacuum, and then, during the vacuum, uses the replica to force +# GlobalVisState->maybe_needed on the primary to move backwards and precede +# the value of OldestXmin set at the beginning of vacuuming the table. + +# Set up nodes +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 'physical'); + +$node_primary->append_conf( + 'postgresql.conf', qq[ +hot_standby_feedback = on +log_recovery_conflict_waits = true +autovacuum = off +log_min_messages = INFO +maintenance_work_mem = 64 +]); +$node_primary->start; + +my $node_replica = PostgreSQL::Test::Cluster->new('standby'); + +$node_primary->backup('my_backup'); +$node_replica->init_from_backup($node_primary, 'my_backup', + has_streaming => 1); + +$node_replica->start; + +my $test_db = "test_db"; +$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db"); + +# Save the original connection info for later use +my $orig_conninfo = $node_primary->connstr(); + +my $table1 = "vac_horizon_floor_table"; + +# Long-running Primary Session A +my $psql_primaryA = + $node_primary->background_psql($test_db, on_error_stop => 1); + +# Long-running Primary Session B +my $psql_primaryB = + $node_primary->background_psql($test_db, on_error_stop => 1); + +# The TIDStore which vacuum uses to store dead items is optimized for its +# target system. On a 32-bit system, our example requires around 9000 rows to +# have enough dead items spread out across enough pages to fill the TIDStore +# and trigger a second round of index vacuuming. We could get away with fewer +# rows on 64-bit systems, but it doesn't seem worth the special case. +my $nrows = 9000; + +# Because vacuum's first pass, pruning, is where we use the GlobalVisState to +# check tuple visibility, GlobalVisState->maybe_needed must move backwards +# during pruning before checking the visibility for a tuple which would have +# been considered HEAPTUPLE_DEAD prior to maybe_needed moving backwards but +# HEAPTUPLE_RECENTLY_DEAD compared to the new, older value of maybe_needed. +# +# We must not only force the horizon on the primary to move backwards but also +# force the vacuuming backend's GlobalVisState to be updated. GlobalVisState +# is forced to update during index vacuuming. +# +# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the +# end of a round of index vacuuming, updating the backend's GlobalVisState +# and, in our case, moving maybe_needed backwards. +# +# Then vacuum's first (pruning) pass will continue and pruning will find our +# later inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to +# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin. +# +# Thus, we must force at least two rounds of index vacuuming to ensure that +# some tuple visibility checks will happen after a round of index vacuuming. +# To accomplish this, we set maintenance_work_mem to its minimum value and +# insert and delete enough rows that we force at least one round of index +# vacuuming before getting to a dead tuple which was killed after the standby +# is disconnected. +$node_primary->safe_psql($test_db, qq[ + CREATE TABLE ${table1}(col1 int) + WITH (autovacuum_enabled=false, fillfactor=10); + INSERT INTO $table1 VALUES(7); + INSERT INTO $table1 SELECT generate_series(1, $nrows) % 3; + CREATE INDEX on ${table1}(col1); + DELETE FROM $table1 WHERE col1 = 0; + INSERT INTO $table1 VALUES(7); +]); + +# We will later move the primary forward while the standby is disconnected. +# For now, however, there is no reason not to wait for the standby to catch +# up. +my $primary_lsn = $node_primary->lsn('flush'); +$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn); + +# Test that the WAL receiver is up and running. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 't'); + +# Set primary_conninfo to something invalid on the replica and reload the +# config. Once the config is reloaded, the startup process will force the WAL +# receiver to restart and it will be unable to reconnect because of the +# invalid connection information. +$node_replica->safe_psql($test_db, qq[ + ALTER SYSTEM SET primary_conninfo = ''; + SELECT pg_reload_conf(); + ]); + +# Wait until the WAL receiver has shut down and been unable to start up again. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 'f'); + +# Now insert and update a tuple which will be visible to the vacuum on the +# primary but which will have xmax newer than the oldest xmin on the standby +# that was recently disconnected. +my $res = $psql_primaryA->query_safe( + qq[ + INSERT INTO $table1 VALUES (99); + UPDATE $table1 SET col1 = 100 WHERE col1 = 99; + SELECT 'after_update'; + ] + ); + +# Make sure the UPDATE finished +like($res, qr/^after_update$/m, "UPDATE occurred on primary session A"); + +# Open a cursor on the primary whose pin will keep VACUUM from getting a +# cleanup lock on the first page of the relation. We want VACUUM to be able to +# start, calculate initial values for OldestXmin and GlobalVisState and then +# be unable to proceed with pruning our dead tuples. This will allow us to +# reconnect the standby and push the horizon back before we start actual +# pruning and vacuuming. +my $primary_cursor1 = "vac_horizon_floor_cursor1"; + +# The first value inserted into the table was a 7, so FETCH FORWARD should +# return a 7. That's how we know the cursor has a pin. +$res = $psql_primaryB->query_safe( + qq[ + BEGIN; + DECLARE $primary_cursor1 CURSOR FOR SELECT * FROM $table1 WHERE col1 = 7; + FETCH $primary_cursor1; + ] + ); + +is($res, 7, qq[Cursor query returned $res. Expected value 7.]); + +# Get the PID of the session which will run the VACUUM FREEZE so that we can +# use it to filter pg_stat_activity later. +my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();"); + +# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs() +# and establish values of OldestXmin and GlobalVisState which are newer than +# all of our dead tuples. Then it will be unable to get a cleanup lock to +# start pruning, so it will hang. +# +# We use VACUUM FREEZE because it will wait for a cleanup lock instead of +# skipping the page pinned by the cursor. Note that works because the target +# tuple's xmax precedes OldestXmin which ensures that lazy_scan_noprune() will +# return false and we will wait for the cleanup lock. +$psql_primaryA->{stdin} .= qq[ + VACUUM (VERBOSE, FREEZE) $table1; + \\echo VACUUM + ]; + +# Make sure the VACUUM command makes it to the server. +$psql_primaryA->{run}->pump_nb(); + +# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is +# just waiting on the lock to start vacuuming. We don't want the standby to +# re-establish a connection to the primary and push the horizon back until +# we've saved initial values in GlobalVisState and calculated OldestXmin. +$node_primary->poll_query_until($test_db, + qq[ + SELECT count(*) >= 1 FROM pg_stat_activity + WHERE pid = $vacuum_pid + AND wait_event = 'BufferPin'; + ], + 't'); + +# Ensure the WAL receiver is still not active on the replica. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 'f'); + +# Allow the WAL receiver connection to re-establish. +$node_replica->safe_psql( + $test_db, qq[ + ALTER SYSTEM SET primary_conninfo = '$orig_conninfo'; + SELECT pg_reload_conf(); + ]); + +# Ensure the new WAL receiver has connected. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 't'); + +# Once the WAL sender is shown on the primary, the replica should have +# connected with the primary and pushed the horizon backward. Primary Session +# A won't see that until the VACUUM FREEZE proceeds and does its first round +# of index vacuuming. +$node_primary->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_replication);] , 't'); + +# Move the cursor forward to the next 7. We inserted the 7 much later, so +# advancing the cursor should allow vacuum to proceed vacuuming most pages of +# the relation. Because we set maintanence_work_mem sufficiently low, we +# expect that a round of index vacuuming has happened and that the vacuum is +# now waiting for the cursor to release its pin on the last page of the +# relation. +$res = $psql_primaryB->query_safe("FETCH $primary_cursor1"); +is($res, 7, + qq[Cursor query returned $res from second fetch. Expected value 7.]); + +# Prevent the test from incorrectly passing by confirming that we did indeed +# do a pass of index vacuuming. +$node_primary->poll_query_until($test_db, qq[ + SELECT index_vacuum_count > 0 + FROM pg_stat_progress_vacuum + WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass; + ] , 't'); + +# Commit the transaction with the open cursor so that the VACUUM can finish. +$psql_primaryB->query_until( + qr/^commit$/m, + qq[ + COMMIT; + \\echo commit + ] + ); + +# VACUUM proceeds with pruning and does a visibility check on each tuple. In +# older versions of Postgres, pruning found our final dead tuple +# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new +# value of maybe_needed. Then heap_prepare_freeze_tuple() would decide the +# tuple xmax should be frozen because it precedes OldestXmin. Vacuum would +# then error out in heap_pre_freeze_checks() with "cannot freeze committed +# xmax". This was fixed by changing pruning to find all +# HEAPTUPLE_RECENTLY_DEAD tuples with xmaxes preceding OldestXmin +# HEAPTUPLE_DEAD and removing them. + +# With the fix, VACUUM should finish successfully, incrementing the table +# vacuum_count. +$node_primary->poll_query_until($test_db, + qq[ + SELECT vacuum_count > 0 + FROM pg_stat_all_tables WHERE relname = '${table1}'; + ] + , 't'); + +$primary_lsn = $node_primary->lsn('flush'); + +# Make sure something causes us to flush +$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);"); + +# Nothing on the replica should cause a recovery conflict, so this should +# finish successfully. +$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn); + +## Shut down psqls +$psql_primaryA->quit; +$psql_primaryB->quit; + +$node_replica->stop(); +$node_primary->stop(); + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> @ 2025-06-17 07:23 ` John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: John Naylor @ 2025-06-17 07:23 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Mon, Jun 16, 2025 at 9:58 PM Melanie Plageman <[email protected]> wrote: > > Test in attached patch seems to do the job on 32 bit and 64 bit when tested. Great! +log_recovery_conflict_waits = true I don't see this on pg16 -- If this is good to have, maybe worth calling out in the commit message as a difference? +# The TIDStore which vacuum uses to store dead items is optimized for its +# target system. On a 32-bit system, our example requires around 9000 rows to +# have enough dead items spread out across enough pages to fill the TIDStore +# and trigger a second round of index vacuuming. We could get away with fewer +# rows on 64-bit systems, but it doesn't seem worth the special case. Minor quibble: I wouldn't say it is deliberately optimized (at least not on local memory) -- it's more of a consequence of pointer-sizes and the somewhat arbitrary choice to set the slab block sizes to hold about X number of chunks. For v19, it might be good to hard-code the block sizes to reduce the possibility of difference and allow a smaller table. +my $nrows = 9000; Running the queries in isolation on an -m32 build shows running 5 index scans, and I found 4000 runs 3 index scans both with and without asserts. Of course, I'm only using the normal 8kB block sizes. In any case, 9000 is already a lot less than 200000, so we can go with that for v17 and v18. -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> @ 2025-06-18 01:04 ` Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2025-06-18 01:04 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Tue, Jun 17, 2025 at 3:23 AM John Naylor <[email protected]> wrote: > > On Mon, Jun 16, 2025 at 9:58 PM Melanie Plageman > <[email protected]> wrote: > > > > Test in attached patch seems to do the job on 32 bit and 64 bit when tested. > > Great! > > +log_recovery_conflict_waits = true > > I don't see this on pg16 -- If this is good to have, maybe worth > calling out in the commit message as a difference? Yea, I'm not 100% sure how useful it is because with hot_standby_feedback on (which we need for the standby to hold the horizon back when it is connected) we shouldn't really see recovery conflicts. I think I added it at some point for debugging issues while writing the test. Perhaps I'll remove it now, as it may be more confusing than anything else. > +# The TIDStore which vacuum uses to store dead items is optimized for its > +# target system. On a 32-bit system, our example requires around 9000 rows to > +# have enough dead items spread out across enough pages to fill the TIDStore > +# and trigger a second round of index vacuuming. We could get away with fewer > +# rows on 64-bit systems, but it doesn't seem worth the special case. > > Minor quibble: I wouldn't say it is deliberately optimized (at least > not on local memory) -- it's more of a consequence of pointer-sizes > and the somewhat arbitrary choice to set the slab block sizes to hold > about X number of chunks. For v19, it might be good to hard-code the > block sizes to reduce the possibility of difference and allow a > smaller table. Thanks for the clarification. I'll clean this comment up in the next version I post after resolving the issue I mention below. > +my $nrows = 9000; > > Running the queries in isolation on an -m32 build shows running 5 > index scans, and I found 4000 runs 3 index scans both with and without > asserts. Of course, I'm only using the normal 8kB block sizes. In any > case, 9000 is already a lot less than 200000, so we can go with that > for v17 and v18. What's odd is that I'm seeing now that I need at least 8000 tuples to get > 1 pass of index vacuuming locally with a 64-bit assert build -- which is more than you are reporting and more than I remember having needed for 64-bit builds when I tested this last year with your patch applied. What's even odder is that I tested on a 32-bit build as well ( -Dc_args='-m32' -Dc_link_args='-m32' --auto-features=disabled) and it doesn't require any more than 8000 tuples to get > 1 pass of index vacuuming. So, currently, on both 32 and 64 bit builds and nrows == 8000, I get 2 passes of index vacuuming. I can't tell what I'm doing wrong. Could you give your full build details? There's no chance that you made a change to the TIDStore that would make it possible for any configuration to have the same size TIDStore on a 32 and 64 bit build, right? - Melanie ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> @ 2025-06-18 04:33 ` John Naylor <[email protected]> 2025-06-20 14:45 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: John Naylor @ 2025-06-18 04:33 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Wed, Jun 18, 2025 at 8:05 AM Melanie Plageman <[email protected]> wrote: > What's odd is that I'm seeing now that I need at least 8000 tuples to > get > 1 pass of index vacuuming locally with a 64-bit assert build -- > which is more than you are reporting and more than I remember having > needed for 64-bit builds when I tested this last year with your patch > applied. > > What's even odder is that I tested on a 32-bit build as well ( > -Dc_args='-m32' -Dc_link_args='-m32' --auto-features=disabled) and it > doesn't require any more than 8000 tuples to get > 1 pass of index > vacuuming. > > So, currently, on both 32 and 64 bit builds and nrows == 8000, I get 2 > passes of index vacuuming. > > I can't tell what I'm doing wrong. Could you give your full build > details? Here's what numbers I'm looking at as well: *** normal meson options: -Dcassert=true -Ddebug=true -Dc_args='-Og' -Dc_args='-fno-omit-frame-pointer' $ meson test -q --print-errorlogs --suite setup && meson test -q --print-errorlogs recovery/048_vacuum_horizon_floor $ grep finished build-debug/testrun/recovery/048_vacuum_horizon_floor/log/048_vacuum_horizon_floor_primary.log 2025-06-18 10:08:19.072 +07 [5730] 048_vacuum_horizon_floor.pl INFO: finished vacuuming "test_db.public.vac_horizon_floor_table": index scans: 12 *** 32-bit meson options: -Dcassert=true -Ddebug=true --auto-features=disabled -Dtap_tests=enabled -Dc_args='-m32' -Dc_link_args='-m32' $ meson test -q --print-errorlogs --suite setup && meson test -q --print-errorlogs recovery/048_vacuum_horizon_floor $ grep finished build-debug32/testrun/recovery/048_vacuum_horizon_floor/log/048_vacuum_horizon_floor_primary.log 2025-06-18 10:27:43.088 +07 [22796] 048_vacuum_horizon_floor.pl INFO: finished vacuuming "test_db.public.vac_horizon_floor_table": index scans: 5 > There's no chance that you made a change to the TIDStore that > would make it possible for any configuration to have the same size > TIDStore on a 32 and 64 bit build, right? Not yet. -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> @ 2025-06-20 14:45 ` Melanie Plageman <[email protected]> 2025-06-23 05:44 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2025-06-20 14:45 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Wed, Jun 18, 2025 at 12:33 AM John Naylor <[email protected]> wrote: > Here's what numbers I'm looking at as well: > > *** normal > meson options: -Dcassert=true -Ddebug=true -Dc_args='-Og' > -Dc_args='-fno-omit-frame-pointer' > > $ meson test -q --print-errorlogs --suite setup && meson test -q > --print-errorlogs recovery/048_vacuum_horizon_floor > > $ grep finished > > build-debug/testrun/recovery/048_vacuum_horizon_floor/log/048_vacuum_horizon_floor_primary.log > > 2025-06-18 10:08:19.072 +07 [5730] 048_vacuum_horizon_floor.pl INFO: > finished vacuuming "test_db.public.vac_horizon_floor_table": index > scans: 12 > Thanks for providing this! I actually did see the same number of index scans as you with 9000 rows. So, I think I figured out why I was seeing the test hang waiting for pg_stat_progress_vacuum to report an index count. Using auto_explain, I determined that the cursor was using an index-only scan with lower row counts. That meant it pinned an index leaf page instead of a heap page and the first round of index vacuuming couldn't complete because btree index vacuuming requires we acquire a cleanup lock on every leaf page. I solved this by disabling all index scans in the cursor's session. I attached the updated patch which passes for me on 32 and 64-bit builds. We've managed to reduce the row count so low (1000-2000 rows) that I'm not sure it matters if we have a 64-bit and 32-bit case. However, since we have the large block comment about the required number of rows, I figured we might as well have the two different nrows. I'll have to do some more research on 14-16 to see if this could be a problem there. I also disabled prefetching, concurrent IO, and read combining for vacuum -- it didn't cause a problem in my local tests, but I could see it interfering with the test and potentially causing flakes/failures on some machines/configurations. That means I'll have to do a slightly different patch for 17 than 18 (17 doesn't have io_combine_limit). Finally, I disabled parallelism as a future-proofing for having heap vacuum parallelism -- wouldn't want a mysterious failure in this test in the future. - Melanie Attachments: [text/x-patch] v2-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch (13.9K, ../../CAAKRu_Z-qjOgUjJqG_ScQFF_E9aEm1+uWPUJKXPwJ0QC27pKOQ@mail.gmail.com/3-v2-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch) download | inline diff: From 8f3496451d82228b846d9cd0e3a1739b44c48440 Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Fri, 13 Jun 2025 09:25:40 -0400 Subject: [PATCH v2] Test that vacuum removes tuples older than OldestXmin If vacuum fails to prune a tuple killed before OldestXmin, it will decide to freeze its xmax and later error out in pre-freeze checks. Add a test reproducing this scenario to the recovery suite which creates a table on a primary, updates the table to generate dead tuples for vacuum, and then, during the vacuum, uses a replica to force GlobalVisState->maybe_needed on the primary to move backwards and precede the value of OldestXmin set at the beginning of vacuuming the table. This test is coverage for a case fixed in 83c39a1f7f3. The test was originally committed to master in aa607980aee but later reverted in efcbb76efe4 due to test instability. The test requires multiple index passes. In Postgres 17+, vacuum uses a TID store for the dead TIDs that is very space efficient. With the old minimum maintenance_work_mem of 1 MB, it required a large number of dead rows to generate enough dead TIDs to force multiple index vacuuming passes. Once the source code changes were made to allow a minimum maintenance_work_mem value of 64kB, the test could be made much faster and more stable. Author: Melanie Plageman <[email protected]> Reviewed-by: Peter Geoghegan <[email protected]> Reviewed-by: John Naylor <[email protected]> Discussion: https://postgr.es/m/CAAKRu_ZJBkidusDut6i%3DbDCiXzJEp93GC1%2BNFaZt4eqanYF3Kw%40mail.gmail.com Backpatch-through: 17 --- src/test/recovery/meson.build | 3 +- .../recovery/t/048_vacuum_horizon_floor.pl | 286 ++++++++++++++++++ 2 files changed, 288 insertions(+), 1 deletion(-) create mode 100644 src/test/recovery/t/048_vacuum_horizon_floor.pl diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 92429d28402..52993c32dbb 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -55,7 +55,8 @@ tests += { 't/044_invalidate_inactive_slots.pl', 't/045_archive_restartpoint.pl', 't/046_checkpoint_logical_slot.pl', - 't/047_checkpoint_physical_slot.pl' + 't/047_checkpoint_physical_slot.pl', + 't/048_vacuum_horizon_floor.pl' ], }, } diff --git a/src/test/recovery/t/048_vacuum_horizon_floor.pl b/src/test/recovery/t/048_vacuum_horizon_floor.pl new file mode 100644 index 00000000000..55b120b0edd --- /dev/null +++ b/src/test/recovery/t/048_vacuum_horizon_floor.pl @@ -0,0 +1,286 @@ +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use Test::More; + +# Test that vacuum prunes away all dead tuples killed before OldestXmin +# +# This test creates a table on a primary, updates the table to generate dead +# tuples for vacuum, and then, during the vacuum, uses the replica to force +# GlobalVisState->maybe_needed on the primary to move backwards and precede +# the value of OldestXmin set at the beginning of vacuuming the table. + +# Set up nodes +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 'physical'); + +# io_combine_limit is set to 1 to avoid pinning more than one buffer at a time +# to ensure determinism in the test. +$node_primary->append_conf( + 'postgresql.conf', qq[ +hot_standby_feedback = on +autovacuum = off +log_min_messages = INFO +maintenance_work_mem = 64 +io_combine_limit = 1 +]); +$node_primary->start; + +my $node_replica = PostgreSQL::Test::Cluster->new('standby'); + +$node_primary->backup('my_backup'); +$node_replica->init_from_backup($node_primary, 'my_backup', + has_streaming => 1); + +$node_replica->start; + +my $test_db = "test_db"; +$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db"); + +# Save the original connection info for later use +my $orig_conninfo = $node_primary->connstr(); + +my $table1 = "vac_horizon_floor_table"; + +# Long-running Primary Session A +my $psql_primaryA = + $node_primary->background_psql($test_db, on_error_stop => 1); + +# Long-running Primary Session B +my $psql_primaryB = + $node_primary->background_psql($test_db, on_error_stop => 1); + +# Our test relies on two rounds of index vacuuming for reasons elaborated +# later. To trigger two rounds of index vacuuming, we must fill up the +# TIDStore with dead items partway through a vacuum of the table. The amount +# of space consumed by dead items in the TIDStore differs based on build +# type. On a 64-bit build, this example (with fillfactor 10) +# requires inserting about 800 rows and deleting a third of them to generate +# enough dead items to fill up the TIDStore with the minimum +# maintenance_work_mem. On a 32-bit build, the required number of rows is +# closer to 1800. We've padded that number to avoid cutting it too +# close. +my $is_64bit = $node_primary->safe_psql($test_db, + qq[SELECT typbyval FROM pg_type WHERE typname = 'int8';]); + +my $nrows = $is_64bit eq 't' ? 1000 : 2000; + +# Because vacuum's first pass, pruning, is where we use the GlobalVisState to +# check tuple visibility, GlobalVisState->maybe_needed must move backwards +# during pruning before checking the visibility for a tuple which would have +# been considered HEAPTUPLE_DEAD prior to maybe_needed moving backwards but +# HEAPTUPLE_RECENTLY_DEAD compared to the new, older value of maybe_needed. +# +# We must not only force the horizon on the primary to move backwards but also +# force the vacuuming backend's GlobalVisState to be updated. GlobalVisState +# is forced to update during index vacuuming. +# +# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the +# end of a round of index vacuuming, updating the backend's GlobalVisState +# and, in our case, moving maybe_needed backwards. +# +# Then vacuum's first (pruning) pass will continue and pruning will find our +# later inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to +# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin. +# +# Thus, we must force at least two rounds of index vacuuming to ensure that +# some tuple visibility checks will happen after a round of index vacuuming. +# To accomplish this, we set maintenance_work_mem to its minimum value and +# insert and delete enough rows that we force at least one round of index +# vacuuming before getting to a dead tuple which was killed after the standby +# is disconnected. +$node_primary->safe_psql($test_db, qq[ + CREATE TABLE ${table1}(col1 int) + WITH (autovacuum_enabled=false, fillfactor=10); + INSERT INTO $table1 VALUES(7); + INSERT INTO $table1 SELECT generate_series(1, $nrows) % 3; + CREATE INDEX on ${table1}(col1); + DELETE FROM $table1 WHERE col1 = 0; + INSERT INTO $table1 VALUES(7); +]); + +# We will later move the primary forward while the standby is disconnected. +# For now, however, there is no reason not to wait for the standby to catch +# up. +my $primary_lsn = $node_primary->lsn('flush'); +$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn); + +# Test that the WAL receiver is up and running. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 't'); + +# Set primary_conninfo to something invalid on the replica and reload the +# config. Once the config is reloaded, the startup process will force the WAL +# receiver to restart and it will be unable to reconnect because of the +# invalid connection information. +$node_replica->safe_psql($test_db, qq[ + ALTER SYSTEM SET primary_conninfo = ''; + SELECT pg_reload_conf(); + ]); + +# Wait until the WAL receiver has shut down and been unable to start up again. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 'f'); + +# Now insert and update a tuple which will be visible to the vacuum on the +# primary but which will have xmax newer than the oldest xmin on the standby +# that was recently disconnected. +my $res = $psql_primaryA->query_safe( + qq[ + INSERT INTO $table1 VALUES (99); + UPDATE $table1 SET col1 = 100 WHERE col1 = 99; + SELECT 'after_update'; + ] + ); + +# Make sure the UPDATE finished +like($res, qr/^after_update$/m, "UPDATE occurred on primary session A"); + +# Open a cursor on the primary whose pin will keep VACUUM from getting a +# cleanup lock on the first page of the relation. We want VACUUM to be able to +# start, calculate initial values for OldestXmin and GlobalVisState and then +# be unable to proceed with pruning our dead tuples. This will allow us to +# reconnect the standby and push the horizon back before we start actual +# pruning and vacuuming. +my $primary_cursor1 = "vac_horizon_floor_cursor1"; + +# The first value inserted into the table was a 7, so FETCH FORWARD should +# return a 7. That's how we know the cursor has a pin. +# Disable index scans so the cursor pins heap pages and not index pages. +$res = $psql_primaryB->query_safe( + qq[ + BEGIN; + SET enable_bitmapscan = off; + SET enable_indexscan = off; + SET enable_indexonlyscan = off; + DECLARE $primary_cursor1 CURSOR FOR SELECT * FROM $table1 WHERE col1 = 7; + FETCH $primary_cursor1; + ] + ); + +is($res, 7, qq[Cursor query returned $res. Expected value 7.]); + +# Get the PID of the session which will run the VACUUM FREEZE so that we can +# use it to filter pg_stat_activity later. +my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();"); + +# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs() +# and establish values of OldestXmin and GlobalVisState which are newer than +# all of our dead tuples. Then it will be unable to get a cleanup lock to +# start pruning, so it will hang. +# +# We use VACUUM FREEZE because it will wait for a cleanup lock instead of +# skipping the page pinned by the cursor. Note that works because the target +# tuple's xmax precedes OldestXmin which ensures that lazy_scan_noprune() will +# return false and we will wait for the cleanup lock. +# +# Disable any prefetching, parallelism, or other concurrent I/O by vacuum. The +# pages of the heap must be processed in order by a single worker to ensure +# test stability (PARALLEL 0 is a future-proofing measure in case we adopt +# parallel heap vacuuming). +$psql_primaryA->{stdin} .= qq[ + SET maintenance_io_concurrency = 0; + VACUUM (VERBOSE, FREEZE, PARALLEL 0) $table1; + \\echo VACUUM + ]; + +# Make sure the VACUUM command makes it to the server. +$psql_primaryA->{run}->pump_nb(); + +# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is +# just waiting on the lock to start vacuuming. We don't want the standby to +# re-establish a connection to the primary and push the horizon back until +# we've saved initial values in GlobalVisState and calculated OldestXmin. +$node_primary->poll_query_until($test_db, + qq[ + SELECT count(*) >= 1 FROM pg_stat_activity + WHERE pid = $vacuum_pid + AND wait_event = 'BufferPin'; + ], + 't'); + +# Ensure the WAL receiver is still not active on the replica. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 'f'); + +# Allow the WAL receiver connection to re-establish. +$node_replica->safe_psql( + $test_db, qq[ + ALTER SYSTEM SET primary_conninfo = '$orig_conninfo'; + SELECT pg_reload_conf(); + ]); + +# Ensure the new WAL receiver has connected. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 't'); + +# Once the WAL sender is shown on the primary, the replica should have +# connected with the primary and pushed the horizon backward. Primary Session +# A won't see that until the VACUUM FREEZE proceeds and does its first round +# of index vacuuming. +$node_primary->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_replication);] , 't'); + +# Move the cursor forward to the next 7. We inserted the 7 much later, so +# advancing the cursor should allow vacuum to proceed vacuuming most pages of +# the relation. Because we set maintanence_work_mem sufficiently low, we +# expect that a round of index vacuuming has happened and that the vacuum is +# now waiting for the cursor to release its pin on the last page of the +# relation. +$res = $psql_primaryB->query_safe("FETCH $primary_cursor1"); +is($res, 7, + qq[Cursor query returned $res from second fetch. Expected value 7.]); + +# Prevent the test from incorrectly passing by confirming that we did indeed +# do a pass of index vacuuming. +$node_primary->poll_query_until($test_db, qq[ + SELECT index_vacuum_count > 0 + FROM pg_stat_progress_vacuum + WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass; + ] , 't'); + +# Commit the transaction with the open cursor so that the VACUUM can finish. +$psql_primaryB->query_until( + qr/^commit$/m, + qq[ + COMMIT; + \\echo commit + ] + ); + +# VACUUM proceeds with pruning and does a visibility check on each tuple. In +# older versions of Postgres, pruning found our final dead tuple +# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new +# value of maybe_needed. Then heap_prepare_freeze_tuple() would decide the +# tuple xmax should be frozen because it precedes OldestXmin. Vacuum would +# then error out in heap_pre_freeze_checks() with "cannot freeze committed +# xmax". This was fixed by changing pruning to find all +# HEAPTUPLE_RECENTLY_DEAD tuples with xmaxes preceding OldestXmin +# HEAPTUPLE_DEAD and removing them. + +# With the fix, VACUUM should finish successfully, incrementing the table +# vacuum_count. +$node_primary->poll_query_until($test_db, + qq[ + SELECT vacuum_count > 0 + FROM pg_stat_all_tables WHERE relname = '${table1}'; + ] + , 't'); + +$primary_lsn = $node_primary->lsn('flush'); + +# Make sure something causes us to flush +$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);"); + +# Nothing on the replica should cause a recovery conflict, so this should +# finish successfully. +$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn); + +## Shut down psqls +$psql_primaryA->quit; +$psql_primaryB->quit; + +$node_replica->stop(); +$node_primary->stop(); + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-20 14:45 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> @ 2025-06-23 05:44 ` John Naylor <[email protected]> 2025-06-23 22:30 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: John Naylor @ 2025-06-23 05:44 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Fri, Jun 20, 2025 at 9:45 PM Melanie Plageman <[email protected]> wrote: > So, I think I figured out why I was seeing the test hang waiting for pg_stat_progress_vacuum to report an index count. > > Using auto_explain, I determined that the cursor was using an index-only scan with lower row counts. That meant it pinned an index leaf page instead of a heap page and the first round of index vacuuming couldn't complete because btree index vacuuming requires we acquire a cleanup lock on every leaf page. > > I solved this by disabling all index scans in the cursor's session. Interesting find! I wondered how it would look if the cursor referred to a extra non-indexed column, but the above seems fine and the whole thing probably easier to reason about with only a single column. > I attached the updated patch which passes for me on 32 and 64-bit builds. We've managed to reduce the row count so low (1000-2000 rows) that I'm not sure it matters if we have a 64-bit and 32-bit case. However, since we have the large block comment about the required number of rows, I figured we might as well have the two different nrows. It seems backwards to have the comment influence the code -- the comment should document decisions around the code. 2000 is already 100x smaller than pg16, so it's not really buying us much to be platform-aware. The first two sentences in the comment seem fine. After that, I'd just say: # We choose the number of rows to make sure we exceed maintenance_work_mem on all platforms we support, but we also want it to be small so that the test runtime is short. Last I checked, our regression tests fail on block sizes other than 8kB. If we ever need to cater to that, this comment covers that eventuality as well. > I'll have to do some more research on 14-16 to see if this could be a problem there. > > I also disabled prefetching, concurrent IO, and read combining for vacuum -- it didn't cause a problem in my local tests, but I could see it interfering with the test and potentially causing flakes/failures on some machines/configurations. That means I'll have to do a slightly different patch for 17 than 18 (17 doesn't have io_combine_limit). > > Finally, I disabled parallelism as a future-proofing for having heap vacuum parallelism -- wouldn't want a mysterious failure in this test in the future. + (PARALLEL 0 is a future-proofing measure in case we adopt + # parallel heap vacuuming) Maybe it's possible to phrase this so it's true regardless of whether we adopt that or not? "PARALLEL 0 shouldn't be necessary, but guards against the possibility of parallel heap vacuuming" -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-20 14:45 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-23 05:44 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> @ 2025-06-23 22:30 ` Melanie Plageman <[email protected]> 2025-06-24 03:49 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: Melanie Plageman @ 2025-06-23 22:30 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Mon, Jun 23, 2025 at 1:44 AM John Naylor <[email protected]> wrote: > > On Fri, Jun 20, 2025 at 9:45 PM Melanie Plageman > <[email protected]> wrote: > > > Using auto_explain, I determined that the cursor was using an index-only scan with lower row counts. That meant it pinned an index leaf page instead of a heap page and the first round of index vacuuming couldn't complete because btree index vacuuming requires we acquire a cleanup lock on every leaf page. > > > > I solved this by disabling all index scans in the cursor's session. > > Interesting find! I wondered how it would look if the cursor referred > to a extra non-indexed column, but the above seems fine and the whole > thing probably easier to reason about with only a single column. Yea, it was a fun little investigation. We could eliminate the possibility of an index-only scan by adding another column and referring to it, but I think forcing a sequential scan explicitly makes the test clearer. > It seems backwards to have the comment influence the code -- the > comment should document decisions around the code. 2000 is already > 100x smaller than pg16, so it's not really buying us much to be > platform-aware. You're right. It is odd to have the location of the comment affect the code :). I've changed it as you suggested. > + (PARALLEL 0 is a future-proofing measure in case we adopt > + # parallel heap vacuuming) > > Maybe it's possible to phrase this so it's true regardless of whether > we adopt that or not? > "PARALLEL 0 shouldn't be necessary, but guards against the possibility > of parallel heap vacuuming" I've adopted this phrasing. Attached v3 has all of the above. I think the only thing that is needed to be changed for the backpatch to 17 is removing io_combine_limit. - Melanie Attachments: [text/x-patch] v3-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch (13.5K, ../../CAAKRu_a-5XA=5W9c=vosypuSqvt8v5RSAn-N__M3nvA9qj-nKg@mail.gmail.com/2-v3-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch) download | inline diff: From 2a3206d2422be11814d3c59f12175138cb18c77d Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Fri, 13 Jun 2025 09:25:40 -0400 Subject: [PATCH v3] Test that vacuum removes tuples older than OldestXmin If vacuum fails to prune a tuple killed before OldestXmin, it will decide to freeze its xmax and later error out in pre-freeze checks. Add a test reproducing this scenario to the recovery suite which creates a table on a primary, updates the table to generate dead tuples for vacuum, and then, during the vacuum, uses a replica to force GlobalVisState->maybe_needed on the primary to move backwards and precede the value of OldestXmin set at the beginning of vacuuming the table. This test is coverage for a case fixed in 83c39a1f7f3. The test was originally committed to master in aa607980aee but later reverted in efcbb76efe4 due to test instability. The test requires multiple index passes. In Postgres 17+, vacuum uses a TID store for the dead TIDs that is very space efficient. With the old minimum maintenance_work_mem of 1 MB, it required a large number of dead rows to generate enough dead TIDs to force multiple index vacuuming passes. Once the source code changes were made to allow a minimum maintenance_work_mem value of 64kB, the test could be made much faster and more stable. Author: Melanie Plageman <[email protected]> Reviewed-by: Peter Geoghegan <[email protected]> Reviewed-by: John Naylor <[email protected]> Discussion: https://postgr.es/m/CAAKRu_ZJBkidusDut6i%3DbDCiXzJEp93GC1%2BNFaZt4eqanYF3Kw%40mail.gmail.com Backpatch-through: 17 --- src/test/recovery/meson.build | 3 +- .../recovery/t/048_vacuum_horizon_floor.pl | 278 ++++++++++++++++++ 2 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 src/test/recovery/t/048_vacuum_horizon_floor.pl diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 28fd9ae8dda..6e78ff1a030 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -54,7 +54,8 @@ tests += { 't/043_no_contrecord_switch.pl', 't/044_invalidate_inactive_slots.pl', 't/045_archive_restartpoint.pl', - 't/047_checkpoint_physical_slot.pl' + 't/047_checkpoint_physical_slot.pl', + 't/048_vacuum_horizon_floor.pl' ], }, } diff --git a/src/test/recovery/t/048_vacuum_horizon_floor.pl b/src/test/recovery/t/048_vacuum_horizon_floor.pl new file mode 100644 index 00000000000..d48a6ef7e0f --- /dev/null +++ b/src/test/recovery/t/048_vacuum_horizon_floor.pl @@ -0,0 +1,278 @@ +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use Test::More; + +# Test that vacuum prunes away all dead tuples killed before OldestXmin +# +# This test creates a table on a primary, updates the table to generate dead +# tuples for vacuum, and then, during the vacuum, uses the replica to force +# GlobalVisState->maybe_needed on the primary to move backwards and precede +# the value of OldestXmin set at the beginning of vacuuming the table. + +# Set up nodes +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 'physical'); + +# io_combine_limit is set to 1 to avoid pinning more than one buffer at a time +# to ensure test determinism. +$node_primary->append_conf( + 'postgresql.conf', qq[ +hot_standby_feedback = on +autovacuum = off +log_min_messages = INFO +maintenance_work_mem = 64 +io_combine_limit = 1 +]); +$node_primary->start; + +my $node_replica = PostgreSQL::Test::Cluster->new('standby'); + +$node_primary->backup('my_backup'); +$node_replica->init_from_backup($node_primary, 'my_backup', + has_streaming => 1); + +$node_replica->start; + +my $test_db = "test_db"; +$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db"); + +# Save the original connection info for later use +my $orig_conninfo = $node_primary->connstr(); + +my $table1 = "vac_horizon_floor_table"; + +# Long-running Primary Session A +my $psql_primaryA = + $node_primary->background_psql($test_db, on_error_stop => 1); + +# Long-running Primary Session B +my $psql_primaryB = + $node_primary->background_psql($test_db, on_error_stop => 1); + +# Our test relies on two rounds of index vacuuming for reasons elaborated +# later. To trigger two rounds of index vacuuming, we must fill up the +# TIDStore with dead items partway through a vacuum of the table. The number +# of rows is just enough to ensure we exceed maintenance_work_mem on all +# supported platforms, while keeping test runtime as short as we can. +my $nrows = 2000; + +# Because vacuum's first pass, pruning, is where we use the GlobalVisState to +# check tuple visibility, GlobalVisState->maybe_needed must move backwards +# during pruning before checking the visibility for a tuple which would have +# been considered HEAPTUPLE_DEAD prior to maybe_needed moving backwards but +# HEAPTUPLE_RECENTLY_DEAD compared to the new, older value of maybe_needed. +# +# We must not only force the horizon on the primary to move backwards but also +# force the vacuuming backend's GlobalVisState to be updated. GlobalVisState +# is forced to update during index vacuuming. +# +# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the +# end of a round of index vacuuming, updating the backend's GlobalVisState +# and, in our case, moving maybe_needed backwards. +# +# Then vacuum's first (pruning) pass will continue and pruning will find our +# later inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to +# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin. +# +# Thus, we must force at least two rounds of index vacuuming to ensure that +# some tuple visibility checks will happen after a round of index vacuuming. +# To accomplish this, we set maintenance_work_mem to its minimum value and +# insert and delete enough rows that we force at least one round of index +# vacuuming before getting to a dead tuple which was killed after the standby +# is disconnected. +$node_primary->safe_psql($test_db, qq[ + CREATE TABLE ${table1}(col1 int) + WITH (autovacuum_enabled=false, fillfactor=10); + INSERT INTO $table1 VALUES(7); + INSERT INTO $table1 SELECT generate_series(1, $nrows) % 3; + CREATE INDEX on ${table1}(col1); + DELETE FROM $table1 WHERE col1 = 0; + INSERT INTO $table1 VALUES(7); +]); + +# We will later move the primary forward while the standby is disconnected. +# For now, however, there is no reason not to wait for the standby to catch +# up. +my $primary_lsn = $node_primary->lsn('flush'); +$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn); + +# Test that the WAL receiver is up and running. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 't'); + +# Set primary_conninfo to something invalid on the replica and reload the +# config. Once the config is reloaded, the startup process will force the WAL +# receiver to restart and it will be unable to reconnect because of the +# invalid connection information. +$node_replica->safe_psql($test_db, qq[ + ALTER SYSTEM SET primary_conninfo = ''; + SELECT pg_reload_conf(); + ]); + +# Wait until the WAL receiver has shut down and been unable to start up again. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 'f'); + +# Now insert and update a tuple which will be visible to the vacuum on the +# primary but which will have xmax newer than the oldest xmin on the standby +# that was recently disconnected. +my $res = $psql_primaryA->query_safe( + qq[ + INSERT INTO $table1 VALUES (99); + UPDATE $table1 SET col1 = 100 WHERE col1 = 99; + SELECT 'after_update'; + ] + ); + +# Make sure the UPDATE finished +like($res, qr/^after_update$/m, "UPDATE occurred on primary session A"); + +# Open a cursor on the primary whose pin will keep VACUUM from getting a +# cleanup lock on the first page of the relation. We want VACUUM to be able to +# start, calculate initial values for OldestXmin and GlobalVisState and then +# be unable to proceed with pruning our dead tuples. This will allow us to +# reconnect the standby and push the horizon back before we start actual +# pruning and vacuuming. +my $primary_cursor1 = "vac_horizon_floor_cursor1"; + +# The first value inserted into the table was a 7, so FETCH FORWARD should +# return a 7. That's how we know the cursor has a pin. +# Disable index scans so the cursor pins heap pages and not index pages. +$res = $psql_primaryB->query_safe( + qq[ + BEGIN; + SET enable_bitmapscan = off; + SET enable_indexscan = off; + SET enable_indexonlyscan = off; + DECLARE $primary_cursor1 CURSOR FOR SELECT * FROM $table1 WHERE col1 = 7; + FETCH $primary_cursor1; + ] + ); + +is($res, 7, qq[Cursor query returned $res. Expected value 7.]); + +# Get the PID of the session which will run the VACUUM FREEZE so that we can +# use it to filter pg_stat_activity later. +my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();"); + +# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs() +# and establish values of OldestXmin and GlobalVisState which are newer than +# all of our dead tuples. Then it will be unable to get a cleanup lock to +# start pruning, so it will hang. +# +# We use VACUUM FREEZE because it will wait for a cleanup lock instead of +# skipping the page pinned by the cursor. Note that works because the target +# tuple's xmax precedes OldestXmin which ensures that lazy_scan_noprune() will +# return false and we will wait for the cleanup lock. +# +# Disable any prefetching, parallelism, or other concurrent I/O by vacuum. The +# pages of the heap must be processed in order by a single worker to ensure +# test stability (PARALLEL 0 shouldn't be necessary but guards against the +# possibility of parallel heap vacuuming). +$psql_primaryA->{stdin} .= qq[ + SET maintenance_io_concurrency = 0; + VACUUM (VERBOSE, FREEZE, PARALLEL 0) $table1; + \\echo VACUUM + ]; + +# Make sure the VACUUM command makes it to the server. +$psql_primaryA->{run}->pump_nb(); + +# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is +# just waiting on the lock to start vacuuming. We don't want the standby to +# re-establish a connection to the primary and push the horizon back until +# we've saved initial values in GlobalVisState and calculated OldestXmin. +$node_primary->poll_query_until($test_db, + qq[ + SELECT count(*) >= 1 FROM pg_stat_activity + WHERE pid = $vacuum_pid + AND wait_event = 'BufferPin'; + ], + 't'); + +# Ensure the WAL receiver is still not active on the replica. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 'f'); + +# Allow the WAL receiver connection to re-establish. +$node_replica->safe_psql( + $test_db, qq[ + ALTER SYSTEM SET primary_conninfo = '$orig_conninfo'; + SELECT pg_reload_conf(); + ]); + +# Ensure the new WAL receiver has connected. +$node_replica->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_wal_receiver);] , 't'); + +# Once the WAL sender is shown on the primary, the replica should have +# connected with the primary and pushed the horizon backward. Primary Session +# A won't see that until the VACUUM FREEZE proceeds and does its first round +# of index vacuuming. +$node_primary->poll_query_until($test_db, qq[ + SELECT EXISTS (SELECT * FROM pg_stat_replication);] , 't'); + +# Move the cursor forward to the next 7. We inserted the 7 much later, so +# advancing the cursor should allow vacuum to proceed vacuuming most pages of +# the relation. Because we set maintanence_work_mem sufficiently low, we +# expect that a round of index vacuuming has happened and that the vacuum is +# now waiting for the cursor to release its pin on the last page of the +# relation. +$res = $psql_primaryB->query_safe("FETCH $primary_cursor1"); +is($res, 7, + qq[Cursor query returned $res from second fetch. Expected value 7.]); + +# Prevent the test from incorrectly passing by confirming that we did indeed +# do a pass of index vacuuming. +$node_primary->poll_query_until($test_db, qq[ + SELECT index_vacuum_count > 0 + FROM pg_stat_progress_vacuum + WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass; + ] , 't'); + +# Commit the transaction with the open cursor so that the VACUUM can finish. +$psql_primaryB->query_until( + qr/^commit$/m, + qq[ + COMMIT; + \\echo commit + ] + ); + +# VACUUM proceeds with pruning and does a visibility check on each tuple. In +# older versions of Postgres, pruning found our final dead tuple +# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new +# value of maybe_needed. Then heap_prepare_freeze_tuple() would decide the +# tuple xmax should be frozen because it precedes OldestXmin. Vacuum would +# then error out in heap_pre_freeze_checks() with "cannot freeze committed +# xmax". This was fixed by changing pruning to find all +# HEAPTUPLE_RECENTLY_DEAD tuples with xmaxes preceding OldestXmin +# HEAPTUPLE_DEAD and removing them. + +# With the fix, VACUUM should finish successfully, incrementing the table +# vacuum_count. +$node_primary->poll_query_until($test_db, + qq[ + SELECT vacuum_count > 0 + FROM pg_stat_all_tables WHERE relname = '${table1}'; + ] + , 't'); + +$primary_lsn = $node_primary->lsn('flush'); + +# Make sure something causes us to flush +$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);"); + +# Nothing on the replica should cause a recovery conflict, so this should +# finish successfully. +$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn); + +## Shut down psqls +$psql_primaryA->quit; +$psql_primaryB->quit; + +$node_replica->stop(); +$node_primary->stop(); + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-20 14:45 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-23 05:44 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-23 22:30 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> @ 2025-06-24 03:49 ` John Naylor <[email protected]> 2025-06-24 13:32 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 0 siblings, 1 reply; 70+ messages in thread From: John Naylor @ 2025-06-24 03:49 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Tue, Jun 24, 2025 at 5:30 AM Melanie Plageman <[email protected]> wrote: > Attached v3 has all of the above. I think the only thing that is > needed to be changed for the backpatch to 17 is removing > io_combine_limit. Looks good to me. -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-20 14:45 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-23 05:44 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-23 22:30 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-24 03:49 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> @ 2025-06-24 13:32 ` Melanie Plageman <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Melanie Plageman @ 2025-06-24 13:32 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Mon, Jun 23, 2025 at 11:49 PM John Naylor <[email protected]> wrote: > > On Tue, Jun 24, 2025 at 5:30 AM Melanie Plageman > <[email protected]> wrote: > > Attached v3 has all of the above. I think the only thing that is > > needed to be changed for the backpatch to 17 is removing > > io_combine_limit. > > Looks good to me. Okay, pushed to 17 (which does have io_combine_limit, I forgot) and master. Now, let's see if the buildfarm is happy. Thanks for all your help! - Melanie ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v25 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 08d4b8e44d7..2b33bf04883 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.47.3 --ecl2euvoqutt27wj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v25-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v20 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 3063abff9a5..0dee1b1a9d8 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 4daa8bef5ee..063a891351a 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.39.5 --3q6txozlgl6t37td Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v20-0004-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v20 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 3063abff9a5..0dee1b1a9d8 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 4daa8bef5ee..063a891351a 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.39.5 --3q6txozlgl6t37td Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v20-0004-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 3063abff9a5..ff34b1b4269 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 4daa8bef5ee..fc3e42e5f70 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.47.1 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v18-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v19 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 3063abff9a5..0dee1b1a9d8 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 4daa8bef5ee..063a891351a 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.39.5 --ipmcfwv2cm3fpxaf Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v19-0004-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v25 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 08d4b8e44d7..2b33bf04883 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.47.3 --ecl2euvoqutt27wj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v25-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 3063abff9a5..ff34b1b4269 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 4daa8bef5ee..fc3e42e5f70 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.47.1 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v18-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v19 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-08-11 13:31 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-08-11 13:31 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 36 ++++++++++++++++++++++++++++-------- src/include/catalog/index.h | 3 +++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 3063abff9a5..0dee1b1a9d8 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,31 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs and needs to be built later on. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1333,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,9 +1342,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, /* * Concurrent build of an index with exclusion constraints is not - * supported. + * supported. If !concurrently, ii_ExclusinOps is currently not needed. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1435,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 4daa8bef5ee..063a891351a 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); -- 2.39.5 --ipmcfwv2cm3fpxaf Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v19-0004-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-12-04 17:20 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-12-04 17:20 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 08d4b8e44d7..cf2d0abf370 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1334,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1327,7 +1345,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1383,9 +1401,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1394,10 +1410,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1435,6 +1454,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1480,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2452,7 +2474,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2512,7 +2535,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index a8033be4bff..9cc94884abc 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -242,7 +242,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -927,7 +928,8 @@ DefineIndex(Oid tableId, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index e2d9e9be41a..c5d5a37f514 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 5473ce9a288..9ff7159ff0c 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v26-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-12-04 17:20 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-12-04 17:20 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 08d4b8e44d7..cf2d0abf370 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1334,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1327,7 +1345,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1383,9 +1401,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1394,10 +1410,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1435,6 +1454,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1480,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2452,7 +2474,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2512,7 +2535,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index a8033be4bff..9cc94884abc 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -242,7 +242,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -927,7 +928,8 @@ DefineIndex(Oid tableId, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index e2d9e9be41a..c5d5a37f514 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 5473ce9a288..9ff7159ff0c 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v26-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-12-09 18:44 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-12-09 18:44 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 08d4b8e44d7..cf2d0abf370 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1334,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1327,7 +1345,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1383,9 +1401,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1394,10 +1410,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1435,6 +1454,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1480,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2452,7 +2474,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2512,7 +2535,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index d9cccb6ac18..d8d8f72a875 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -242,7 +242,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -927,7 +928,8 @@ DefineIndex(Oid tableId, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index e2d9e9be41a..c5d5a37f514 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 5473ce9a288..9ff7159ff0c 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v27-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-12-09 18:44 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-12-09 18:44 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 08d4b8e44d7..cf2d0abf370 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1334,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1327,7 +1345,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1383,9 +1401,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1394,10 +1410,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1435,6 +1454,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1480,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2452,7 +2474,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2512,7 +2535,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index d9cccb6ac18..d8d8f72a875 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -242,7 +242,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -927,7 +928,8 @@ DefineIndex(Oid tableId, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index e2d9e9be41a..c5d5a37f514 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 5473ce9a288..9ff7159ff0c 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v27-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-12-13 18:27 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-12-13 18:27 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/cluster.c | 8 ++--- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 6 files changed, 57 insertions(+), 27 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index bd77584bc99..60efa77d34f 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1334,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1327,7 +1345,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1383,9 +1401,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1394,10 +1410,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1435,6 +1454,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1480,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2452,7 +2474,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2512,7 +2535,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 7f772c5c4f8..a2d72ce494d 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -70,8 +70,7 @@ typedef struct static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid, Oid userid, int options); -static void rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose); +static void rebuild_relation(Relation OldHeap, Relation index, bool verbose); static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, MultiXactId *pCutoffMulti); @@ -415,7 +414,7 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid, TransferPredicateLocksToHeapRelation(OldHeap); /* rebuild_relation does all the dirty work */ - rebuild_relation(cmd, OldHeap, index, verbose); + rebuild_relation(OldHeap, index, verbose); /* rebuild_relation closes OldHeap, and index if valid */ out: @@ -629,8 +628,7 @@ mark_index_clustered(Relation rel, Oid indexOid, bool is_internal) * On exit, they are closed, but locks on them are not released. */ static void -rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose) +rebuild_relation(Relation OldHeap, Relation index, bool verbose) { Oid tableOid = RelationGetRelid(OldHeap); Oid accessMethod = OldHeap->rd_rel->relam; diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index d9cccb6ac18..d8d8f72a875 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -242,7 +242,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -927,7 +928,8 @@ DefineIndex(Oid tableId, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index e2d9e9be41a..c5d5a37f514 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 5473ce9a288..9ff7159ff0c 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v28-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2025-12-13 18:27 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2025-12-13 18:27 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/cluster.c | 8 ++--- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 6 files changed, 57 insertions(+), 27 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index bd77584bc99..60efa77d34f 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1290,15 +1290,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1317,6 +1334,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1327,7 +1345,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1383,9 +1401,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1394,10 +1410,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1435,6 +1454,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1458,7 +1480,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2452,7 +2474,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2512,7 +2535,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 7f772c5c4f8..a2d72ce494d 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -70,8 +70,7 @@ typedef struct static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid, Oid userid, int options); -static void rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose); +static void rebuild_relation(Relation OldHeap, Relation index, bool verbose); static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, MultiXactId *pCutoffMulti); @@ -415,7 +414,7 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid, TransferPredicateLocksToHeapRelation(OldHeap); /* rebuild_relation does all the dirty work */ - rebuild_relation(cmd, OldHeap, index, verbose); + rebuild_relation(OldHeap, index, verbose); /* rebuild_relation closes OldHeap, and index if valid */ out: @@ -629,8 +628,7 @@ mark_index_clustered(Relation rel, Oid indexOid, bool is_internal) * On exit, they are closed, but locks on them are not released. */ static void -rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose) +rebuild_relation(Relation OldHeap, Relation index, bool verbose) { Oid tableOid = RelationGetRelid(OldHeap); Oid accessMethod = OldHeap->rd_rel->relam; diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index d9cccb6ac18..d8d8f72a875 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -242,7 +242,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -927,7 +928,8 @@ DefineIndex(Oid tableId, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index e2d9e9be41a..c5d5a37f514 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index dda95e54903..4bf909078d8 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 5473ce9a288..9ff7159ff0c 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v28-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-08 16:47 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-08 16:47 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/cluster.c | 8 ++--- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 6 files changed, 57 insertions(+), 27 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 06f6dfc37a5..094f3d36047 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -70,8 +70,7 @@ typedef struct static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid, Oid userid, int options); -static void rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose); +static void rebuild_relation(Relation OldHeap, Relation index, bool verbose); static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, MultiXactId *pCutoffMulti); @@ -415,7 +414,7 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid, TransferPredicateLocksToHeapRelation(OldHeap); /* rebuild_relation does all the dirty work */ - rebuild_relation(cmd, OldHeap, index, verbose); + rebuild_relation(OldHeap, index, verbose); /* rebuild_relation closes OldHeap, and index if valid */ out: @@ -629,8 +628,7 @@ mark_index_clustered(Relation rel, Oid indexOid, bool is_internal) * On exit, they are closed, but locks on them are not released. */ static void -rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose) +rebuild_relation(Relation OldHeap, Relation index, bool verbose) { Oid tableOid = RelationGetRelid(OldHeap); Oid accessMethod = OldHeap->rd_rel->relam; diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 08c86cc163c..e837c308e27 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v29-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-08 16:47 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-08 16:47 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/cluster.c | 8 ++--- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 6 files changed, 57 insertions(+), 27 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 06f6dfc37a5..094f3d36047 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -70,8 +70,7 @@ typedef struct static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid, Oid userid, int options); -static void rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose); +static void rebuild_relation(Relation OldHeap, Relation index, bool verbose); static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, MultiXactId *pCutoffMulti); @@ -415,7 +414,7 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid, TransferPredicateLocksToHeapRelation(OldHeap); /* rebuild_relation does all the dirty work */ - rebuild_relation(cmd, OldHeap, index, verbose); + rebuild_relation(OldHeap, index, verbose); /* rebuild_relation closes OldHeap, and index if valid */ out: @@ -629,8 +628,7 @@ mark_index_clustered(Relation rel, Oid indexOid, bool is_internal) * On exit, they are closed, but locks on them are not released. */ static void -rebuild_relation(RepackCommand cmd, - Relation OldHeap, Relation index, bool verbose) +rebuild_relation(Relation OldHeap, Relation index, bool verbose) { Oid tableOid = RelationGetRelid(OldHeap); Oid accessMethod = OldHeap->rd_rel->relam; diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 08c86cc163c..e837c308e27 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v29-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-12 16:30 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-12 16:30 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v30-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-12 16:30 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-12 16:30 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v30-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-22 08:23 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-22 08:23 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v31-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-22 08:23 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-22 08:23 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v31-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v33 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-27 10:48 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-27 10:48 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --r7amdyq3rmktb267 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v33-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-27 10:48 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-27 10:48 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v32-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v33 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-27 10:48 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-27 10:48 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --r7amdyq3rmktb267 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v33-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-01-27 10:48 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-01-27 10:48 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v32-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v35 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-02-16 19:49 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-02-16 19:49 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --kmtyk4ln3767qrma Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v35-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-02-16 19:49 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-02-16 19:49 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v34-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v35 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-02-16 19:49 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-02-16 19:49 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --kmtyk4ln3767qrma Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v35-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-t.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-02-16 19:49 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-02-16 19:49 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v34-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-02-27 18:01 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-02-27 18:01 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v36-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-02-27 18:01 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-02-27 18:01 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v36-0003-Move-conversion-of-a-historic-to-MVCC-snapshot-to-a-.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-02 14:30 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-02 14:30 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v37-0003-Add-CONCURRENTLY-option-to-REPACK-command.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-02 14:30 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-02 14:30 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v37-0003-Add-CONCURRENTLY-option-to-REPACK-command.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-05 18:58 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-05 18:58 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v38-0003-Add-CONCURRENTLY-option-to-REPACK-command.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-05 18:58 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-05 18:58 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v38-0003-Add-CONCURRENTLY-option-to-REPACK-command.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-09 10:35 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-09 10:35 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v39-0003-Add-CONCURRENTLY-option-to-REPACK-command.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v40 1/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-09 10:35 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-09 10:35 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --k4rn5ob3nx2ufkij Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v40-0002-Add-CONCURRENTLY-option-to-REPACK-command.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v40 1/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-09 10:35 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-09 10:35 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --k4rn5ob3nx2ufkij Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v40-0002-Add-CONCURRENTLY-option-to-REPACK-command.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-09 10:35 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-09 10:35 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v39-0003-Add-CONCURRENTLY-option-to-REPACK-command.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-11 13:15 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-11 13:15 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 8b3c60d91f9..544dd0d6e3d 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1291,15 +1291,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1318,6 +1335,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1328,7 +1346,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1384,9 +1402,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1395,10 +1411,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1436,6 +1455,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1459,7 +1481,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2453,7 +2475,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2513,7 +2536,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cbd76066f74..4b37d92a49e 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -244,7 +244,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -931,7 +932,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 3cd35c5c457..8d23aa917e5 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 36b70689254..144e241ebd3 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -105,6 +105,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index bf54d39feb0..40ec249a7a1 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0002-Do-not-dereference-varattrib_4b-in-VARSIZE_4B.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 1/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-11 13:15 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-11 13:15 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v41-0002-Add-va_header-field-to-the-varattrib_4b-union.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v43 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-11 13:15 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-11 13:15 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index d8219b18c48..98d38dc5229 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1291,15 +1291,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1318,6 +1335,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1328,7 +1346,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1384,9 +1402,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1395,10 +1411,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1436,6 +1455,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1459,7 +1481,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2453,7 +2475,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2513,7 +2536,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index b89c6855364..836eedb8ed5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -244,7 +244,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -931,7 +932,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 3cd35c5c457..8d23aa917e5 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 36b70689254..144e241ebd3 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -105,6 +105,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index bf54d39feb0..40ec249a7a1 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0002-Do-not-dereference-varattrib_4b-in-VARSIZE_4B.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH v43 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-11 13:15 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-11 13:15 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index d8219b18c48..98d38dc5229 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1291,15 +1291,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1318,6 +1335,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1328,7 +1346,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1384,9 +1402,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1395,10 +1411,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1436,6 +1455,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1459,7 +1481,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2453,7 +2475,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2513,7 +2536,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index b89c6855364..836eedb8ed5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -244,7 +244,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -931,7 +932,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 3cd35c5c457..8d23aa917e5 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 36b70689254..144e241ebd3 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -105,6 +105,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index bf54d39feb0..40ec249a7a1 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0002-Do-not-dereference-varattrib_4b-in-VARSIZE_4B.patch" ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-11 13:15 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-11 13:15 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 8b3c60d91f9..544dd0d6e3d 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1291,15 +1291,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1318,6 +1335,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1328,7 +1346,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1384,9 +1402,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1395,10 +1411,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1436,6 +1455,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1459,7 +1481,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2453,7 +2475,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2513,7 +2536,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cbd76066f74..4b37d92a49e 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -244,7 +244,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -931,7 +932,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 3cd35c5c457..8d23aa917e5 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 36b70689254..144e241ebd3 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -105,6 +105,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index bf54d39feb0..40ec249a7a1 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0002-Do-not-dereference-varattrib_4b-in-VARSIZE_4B.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
* [PATCH 1/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). @ 2026-03-11 13:15 Antonin Houska <[email protected]> 0 siblings, 0 replies; 70+ messages in thread From: Antonin Houska @ 2026-03-11 13:15 UTC (permalink / raw) This patch moves the code to index_create_copy() and adds a "concurrently" parameter so it can be used by REPACK (CONCURRENTLY). With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. --- src/backend/catalog/index.c | 54 +++++++++++++++++++++++--------- src/backend/commands/indexcmds.c | 6 ++-- src/backend/nodes/makefuncs.c | 9 +++--- src/include/catalog/index.h | 3 ++ src/include/nodes/makefuncs.h | 4 ++- 5 files changed, 54 insertions(+), 22 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 5ee6389d39c..f8e6c3d804e 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1288,15 +1288,32 @@ index_create(Relation heapRelation, /* * index_concurrently_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. - * - * "tablespaceOid" is the tablespace to use for this index. + * Variant of index_create_copy(), called during concurrent reindex + * processing. */ Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName) +{ + return index_create_copy(heapRelation, oldIndexId, tablespaceOid, newName, + true); +} + +/* + * index_create_copy + * + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on, otherwise it's built immediately. + * + * "tablespaceOid" is the tablespace to use for this index. + * + * The actual implementation of index_concurrently_create_copy(), reusable for + * other purposes. + */ +Oid +index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, + const char *newName, bool concurrently) { Relation indexRelation; IndexInfo *oldInfo, @@ -1315,6 +1332,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1325,7 +1343,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1381,9 +1399,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1392,10 +1408,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1433,6 +1452,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1456,7 +1478,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2450,7 +2472,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2510,7 +2533,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 635679cc1f2..34209bd1393 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -243,7 +243,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -930,7 +931,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 2caec621d73..ca7e21e8349 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index b259c4141ed..3426087b445 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -99,6 +99,9 @@ extern Oid index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, const char *newName); +extern Oid index_create_copy(Relation heapRelation, Oid oldIndexId, + Oid tablespaceOid, const char *newName, + bool concurrently); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 982ec25ae14..dcea148ae1a 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v41-0002-Add-va_header-field-to-the-varattrib_4b-union.patch ^ permalink raw reply [nested|flat] 70+ messages in thread
end of thread, other threads:[~2026-03-11 13:15 UTC | newest] Thread overview: 70+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-08-31 22:29 Re: Eliminate redundant tuple visibility check in vacuum Melanie Plageman <[email protected]> 2023-09-06 17:04 ` Robert Haas <[email protected]> 2023-09-06 21:21 ` Melanie Plageman <[email protected]> 2023-09-07 17:37 ` Robert Haas <[email protected]> 2023-09-07 19:09 ` Melanie Plageman <[email protected]> 2023-09-07 19:29 ` Robert Haas <[email protected]> 2023-09-07 22:23 ` Melanie Plageman <[email protected]> 2023-09-08 15:06 ` Robert Haas <[email protected]> 2023-09-11 12:04 ` Melanie Plageman <[email protected]> 2023-09-13 17:29 ` Andres Freund <[email protected]> 2023-09-21 19:53 ` Robert Haas <[email protected]> 2023-09-21 20:07 ` Robert Haas <[email protected]> 2023-09-28 12:46 ` Melanie Plageman <[email protected]> 2023-09-28 15:25 ` Robert Haas <[email protected]> 2025-06-12 08:10 Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-12 22:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-13 14:12 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-16 14:58 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-17 07:23 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-18 01:04 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-18 04:33 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-20 14:45 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-23 05:44 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-23 22:30 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-06-24 03:49 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin John Naylor <[email protected]> 2025-06-24 13:32 ` Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin Melanie Plageman <[email protected]> 2025-08-11 13:31 [PATCH v25 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH v20 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH v20 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH v19 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH v25 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-08-11 13:31 [PATCH v19 3/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-12-04 17:20 [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-12-04 17:20 [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-12-09 18:44 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-12-09 18:44 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-12-13 18:27 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2025-12-13 18:27 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-08 16:47 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-08 16:47 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-12 16:30 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-12 16:30 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-22 08:23 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-22 08:23 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-27 10:48 [PATCH v33 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-27 10:48 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-27 10:48 [PATCH v33 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-01-27 10:48 [PATCH 2/6] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-02-16 19:49 [PATCH v35 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-02-16 19:49 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-02-16 19:49 [PATCH v35 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-02-16 19:49 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-02-27 18:01 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-02-27 18:01 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-02 14:30 [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-02 14:30 [PATCH 2/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-05 18:58 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-05 18:58 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-09 10:35 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-09 10:35 [PATCH v40 1/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-09 10:35 [PATCH v40 1/4] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-09 10:35 [PATCH 2/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-11 13:15 [PATCH 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-11 13:15 [PATCH 1/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-11 13:15 [PATCH v43 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-11 13:15 [PATCH v43 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-11 13:15 [PATCH 1/7] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-03-11 13:15 [PATCH 1/5] Refactor index_concurrently_create_copy() for use with REPACK (CONCURRENTLY). Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox