pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feed[WIP] [B-Tree] Retail IndexTuple deletion
47+ messages / 10 participants
[nested] [flat]
* [WIP] [B-Tree] Retail IndexTuple deletion
@ 2018-06-18 04:39 Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 19:59 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-18 04:39 UTC (permalink / raw)
To: pgsql-hackers
Hi,
I have written a code for quick indextuple deletion from an relation by
heap tuple TID. The code relate to "Retail IndexTuple deletion"
enhancement of btree index on postgresql wiki [1].
Briefly, it includes three steps:
1. Key generation for index tuple searching.
2. Index relation search for tuple with the heap tuple TID.
3. Deletion of the tuple from the index relation.
Now, index relation cleaning performs by vacuum which scan all index
relation for dead entries sequentially, tuple-by-tuple. This simplistic
and safe method can be significantly surpasses in the cases, than a
number of dead entries is not large by retail deletions which uses index
scans for search dead entries. Also, it can be used by distributed
systems for reduce cost of a global index vacuum.
Patch '0001-retail-indextuple-deletion' introduce new function
amtargetdelete() in access method interface. Patch
'0002-quick-vacuum-strategy' implements this function for an alternative
strategy of lazy index vacuum, called 'Quick Vacuum'.
The code demands hold DEAD tuple storage until scan key will be created.
In this version I add 'target_index_deletion_factor' option. If it more
than 0, heap_page_prune() uses ItemIdMarkDead() instead of
ItemIdSetDead() function for set DEAD flag and hold the tuple storage.
Next step is developing background worker which will collect pairs (tid,
scankey) of DEAD tuples from heap_page_prune() function.
Here are test description and some execution time measurements results
showing the benefit of this patches:
Test:
-----
create table test_range(id serial primary key, value integer);
insert into test_range (value) select random()*1e7/10^N from
generate_series(1, 1e7);
DELETE FROM test_range WHERE value=1;
VACUUM test_range;
Results:
--------
| n | t1, s | t2, s | speedup |
|---|---------------------------|
| 0 | 0.00003| 0.4476 | 1748.4 |
| 1 | 0.00006| 0.5367 | 855.99 |
| 2 | 0.0004 | 0.9804 | 233.99 |
| 3 | 0.0048 | 1.6493 | 34.576 |
| 4 | 0.5600 | 2.4854 | 4.4382 |
| 5 | 3.3300 | 3.9555 | 1.2012 |
| 6 | 17.700 | 5.6000 | 0.3164 |
|---|---------------------------|
in the table, t1 - measured execution time of lazy_vacuum_index()
function by Quick-Vacuum strategy; t2 - measured execution time of
lazy_vacuum_index() function by Lazy-Vacuum strategy;
Note, guaranteed allowable time of index scans (used for quick deletion)
will be achieved by storing equal-key index tuples in physical TID order
[2] with patch [3].
[1]
https://wiki.postgresql.org/wiki/Key_normalization#Retail_IndexTuple_deletion
[2]
https://wiki.postgresql.org/wiki/Key_normalization#Making_all_items_in_the_index_unique_by_treating_...
[3]
https://www.postgresql.org/message-id/CAH2-WzkVb0Kom%3DR%2B88fDFb%3DJSxZMFvbHVC6Mn9LJ2n%3DX%3DkS-Uw%...
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
Attachments: download all (2 files, .tar.gz)
0001-retail-indextuple-deletion.patch (text/x-patch, 12.0K) download
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3..96f1d47 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbc..a0e06bd 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182..acf14e7 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42eff..d7a53d2 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df3..5fb32d6 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 22b5cc9..a25d844 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -730,6 +730,23 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexBulkDeleteResult *
+index_target_delete(IndexVacuumInfo *info,
+ IndexBulkDeleteResult *stats,
+ Relation hrel,
+ Datum *values,
+ bool *isnull,
+ ItemPointer htid)
+{
+ Relation indexRelation = info->index;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, hrel, values, isnull, htid);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 27a3032..dd87a05 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -34,7 +34,9 @@
#include "storage/smgr.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/syscache.h"
/* Working state needed by btvacuumpage */
@@ -127,6 +129,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -884,6 +887,195 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+static void
+cleanup_block(IndexVacuumInfo *info,
+ IndexBulkDeleteResult *stats, BlockNumber blkno)
+{
+ BTVacState vstate;
+
+ vstate.info = info;
+ vstate.stats = stats;
+ vstate.callback = NULL;
+ vstate.callback_state = NULL;
+ vstate.cycleid = 0;
+ vstate.lastBlockVacuumed = BTREE_METAPAGE; /* Initialise at first block */
+ vstate.lastBlockLocked = BTREE_METAPAGE;
+ vstate.totFreePages = 0;
+ vstate.oldestBtpoXact = InvalidTransactionId;
+
+ /* Create a temporary memory context to run _bt_pagedel in */
+ vstate.pagedelcontext = AllocSetContextCreate(CurrentMemoryContext,
+ "_bt_pagedel",
+ ALLOCSET_DEFAULT_SIZES);
+ btvacuumpage(&vstate, blkno, blkno);
+
+ if (vstate.totFreePages > 0)
+ IndexFreeSpaceMapVacuum(info->index);
+
+ stats->pages_free += vstate.totFreePages;
+
+ MemoryContextDelete(vstate.pagedelcontext);
+}
+
+/*
+ * Build scan key for an index relation by a heap tuple TID
+ */
+static int
+build_scan_key(ScanKey skey, Relation rel, Relation irel, Datum *values, bool *nulls)
+{
+ TupleDesc desc = RelationGetDescr(rel);
+ HeapTuple tuple;
+ Datum indclassDatum;
+ int attoff;
+ oidvector* opclass;
+ int2vector* indkey = &irel->rd_index->indkey;
+ bool isnull;
+
+ tuple = heap_form_tuple(desc, values, nulls);
+ indclassDatum = SysCacheGetAttr(INDEXRELID, irel->rd_indextuple,
+ Anum_pg_index_indclass, &isnull);
+
+ opclass = (oidvector *) DatumGetPointer(indclassDatum);
+ for (attoff = 0; attoff < IndexRelationGetNumberOfKeyAttributes(irel); attoff++)
+ {
+ Oid operator;
+ Oid opfamily;
+ RegProcedure regop;
+ int pkattno = attoff + 1;
+ int mainattno = indkey->values[attoff];
+ Oid optype = get_opclass_input_type(opclass->values[attoff]);
+ Datum scanvalue;
+ StrategyNumber strategy;
+
+ if (mainattno > 0)
+ scanvalue = heap_getattr(tuple, mainattno, desc, &isnull);
+ else
+ return -1;
+
+ /*
+ * Load the operator info. We need this to get the equality operator
+ * function for the scan key.
+ */
+ opfamily = get_opclass_family(opclass->values[attoff]);
+
+ if (irel->rd_exclstrats != NULL)
+ strategy = irel->rd_exclstrats[attoff];
+ else
+ strategy = BTEqualStrategyNumber;
+
+ operator = get_opfamily_member(opfamily, optype, optype, strategy);
+ if (!OidIsValid(operator))
+ return -1;
+
+ regop = get_opcode(operator);
+
+ /* Initialize the scan key. */
+ ScanKeyInit(&skey[attoff],
+ pkattno,
+ strategy,
+ regop,
+ scanvalue);
+
+ /* Check for null value. */
+ if (isnull)
+ skey[attoff].sk_flags |= SK_ISNULL;
+
+ /* Use index collation */
+ if (irel->rd_indcollation != NULL)
+ skey[attoff].sk_collation = irel->rd_indcollation[attoff];
+
+ /* Use index procedure */
+ if (irel->rd_exclprocs != NULL)
+ fmgr_info(irel->rd_exclprocs[attoff], &(skey[attoff].sk_func));
+ }
+
+ return 0;
+}
+
+IndexBulkDeleteResult*
+bttargetdelete(IndexVacuumInfo *info,
+ IndexBulkDeleteResult *stats,
+ Relation hrel,
+ Datum *values,
+ bool *isnull,
+ ItemPointer htid)
+{
+ Relation irel = info->index;
+ ScanKeyData skey[INDEX_MAX_KEYS];
+ SnapshotData snap;
+ IndexScanDesc scan;
+ ItemPointer tid;
+ bool found = false;
+ OffsetNumber offnum;
+ BlockNumber blkno;
+
+ if (stats == NULL)
+ {
+ stats = (IndexBulkDeleteResult *) palloc0(sizeof(IndexBulkDeleteResult));
+ stats->estimated_count = false;
+ stats->num_index_tuples = 0;
+ stats->pages_deleted = 0;
+ }
+
+ if (build_scan_key(skey, hrel, irel, values, isnull) != 0)
+ {
+ pfree(stats);
+ return NULL;
+ }
+
+ InitDirtySnapshot(snap);
+ scan = index_beginscan(hrel, irel, &snap, IndexRelationGetNumberOfKeyAttributes(irel), 0);
+ index_rescan(scan, skey, IndexRelationGetNumberOfKeyAttributes(irel), NULL, 0);
+
+ /* Pass along index entries array with equal scan key value */
+ while ((!found) && ((tid = index_getnext_tid(scan, ForwardScanDirection)) != NULL))
+ {
+ if (ItemPointerEquals(tid, htid))
+ {
+ BTScanPos idx_pos = (BTScanPos) &(((BTScanOpaque) scan->opaque)->currPos);
+ blkno = idx_pos->currPage;
+ offnum = idx_pos->items[idx_pos->itemIndex].indexOffset;
+ found = true;
+ }
+ }
+ index_endscan(scan);
+
+ /* Delete index tuple */
+ if (found)
+ {
+ Buffer buf;
+ Page page;
+ bool needLock;
+ int npages;
+ IndexTuple itup;
+
+ needLock = !RELATION_IS_LOCAL(irel);
+ if (needLock)
+ LockRelationForExtension(irel, ExclusiveLock);
+ npages = RelationGetNumberOfBlocks(irel);
+ if (needLock)
+ UnlockRelationForExtension(irel, ExclusiveLock);
+ if (blkno >= npages)
+ return stats;
+
+ buf = ReadBuffer(irel, blkno);
+ page = BufferGetPage(buf);
+ LockBufferForCleanup(buf);
+
+ itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
+
+ if (ItemPointerEquals(htid, &(itup->t_tid)))
+ {
+ _bt_delitems_vacuum(irel, buf, &offnum, 1, blkno);
+ stats->tuples_removed++;
+ stats->num_pages = npages;
+ }
+ _bt_relbuf(irel, buf);
+ cleanup_block(info, stats, blkno);
+ }
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4a9b5da..3257281 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -56,6 +56,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6..411bb43 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,13 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexBulkDeleteResult *(*amtargetdelete_function) (IndexVacuumInfo *info,
+ IndexBulkDeleteResult *stats,
+ Relation hrel,
+ Datum *values,
+ bool *isnull,
+ ItemPointer htid);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +214,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 24c720b..cdcc6db 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -163,6 +163,12 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexBulkDeleteResult *index_target_delete(IndexVacuumInfo *info,
+ IndexBulkDeleteResult *stats,
+ Relation hrel,
+ Datum *values,
+ bool *isnull,
+ ItemPointer htid);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 04ecb4c..0de5a24 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -508,6 +508,12 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexBulkDeleteResult *bttargetdelete(IndexVacuumInfo *info,
+ IndexBulkDeleteResult *stats,
+ Relation hrel,
+ Datum *values,
+ bool *isnull,
+ ItemPointer htid);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
0002-quick-vacuum-strategy.patch (text/x-patch, 7.4K) download
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343..d394a7a 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -580,8 +583,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
ndeleted++;
+ if (chainitems[i] == latestdead)
+ continue;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +603,24 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ ItemIdSetDeadRedirect(rootlp, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
- else
+ }
+ else {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -706,7 +726,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 5649a70..04127a4 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -137,9 +137,9 @@ typedef struct LVRelStats
int num_index_scans;
TransactionId latestRemovedXid;
bool lock_waiter_detected;
+ Relation rel;
} LVRelStats;
-
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
@@ -519,6 +519,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
vacrelstats->tupcount_pages = 0;
vacrelstats->nonempty_pages = 0;
vacrelstats->latestRemovedXid = InvalidTransactionId;
+ vacrelstats->rel = onerel;
lazy_space_alloc(vacrelstats, nblocks);
frozen = palloc(sizeof(xl_heap_freeze_tuple) * MaxHeapTuplesPerPage);
@@ -1671,6 +1672,47 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Get tuple from heap for a scan key building. If tuples in HOT chain
+ * have not a storage, return NULL.
+ */
+static HeapTuple get_tuple_by_tid(Relation rel, ItemPointer tid)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ ItemId lp;
+ HeapTuple tuple;
+
+ buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
+ LockBuffer(buffer, BUFFER_LOCK_SHARE);
+
+ page = (Page) BufferGetPage(buffer);
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
+
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
+ }
+
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(rel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ UnlockReleaseBuffer(buffer);
+ return tuple;
+}
+
+#include "access/nbtree.h"
/*
* lazy_vacuum_index() -- vacuum one index relation.
@@ -1684,6 +1726,7 @@ lazy_vacuum_index(Relation indrel,
LVRelStats *vacrelstats)
{
IndexVacuumInfo ivinfo;
+ bool use_target_strategy = true;//(vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
PGRUsage ru0;
pg_rusage_init(&ru0);
@@ -1696,9 +1739,33 @@ lazy_vacuum_index(Relation indrel,
ivinfo.num_heap_tuples = vacrelstats->old_live_tuples;
ivinfo.strategy = vac_strategy;
- /* Do bulk deletion */
- *stats = index_bulk_delete(&ivinfo, *stats,
- lazy_tid_reaped, (void *) vacrelstats);
+ if ((use_target_strategy) && (indrel->rd_amroutine->amtargetdelete != NULL))
+ {
+ int tnum;
+ TupleDesc desc = RelationGetDescr(vacrelstats->rel);
+ Datum* values = (Datum *) palloc(desc->natts * sizeof(Datum));
+ bool* isnull = (bool *) palloc(desc->natts * sizeof(bool));
+
+ for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ {
+ HeapTuple tuple = get_tuple_by_tid(vacrelstats->rel, &(vacrelstats->dead_tuples[tnum]));
+
+ if (tuple == NULL)
+ continue;
+
+ heap_deform_tuple(tuple, desc, values, isnull);
+ *stats = index_target_delete(&ivinfo, *stats, vacrelstats->rel, values, isnull, &(vacrelstats->dead_tuples[tnum]));
+ if (*stats == NULL)
+ break;
+ }
+ pfree(values);
+ pfree(isnull);
+ }
+
+ if ((!use_target_strategy) || (*stats == NULL))
+ /* Do bulk deletion */
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_tid_reaped, (void *) vacrelstats);
ereport(elevel,
(errmsg("scanned index \"%s\" to remove %d row versions",
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fa3c8a7..3f6c83b 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3236,7 +3236,15 @@ static struct config_real ConfigureNamesReal[] =
0.1, 0.0, 100.0,
NULL, NULL, NULL
},
-
+ {
+ {"target_index_deletion_factor", PGC_SIGHUP, AUTOVACUUM,
+ gettext_noop("Maximum number of vacuumed tuples as a fraction of reltuples where we can use target index vacuum strategy."),
+ NULL
+ },
+ &target_index_deletion_factor,
+ 0.01, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
{
{"checkpoint_completion_target", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval."),
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7..c5a7bc8 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,7 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
/*
* HeapScanIsValid
* True iff the heap scan is valid.
diff --git a/src/include/storage/itemid.h b/src/include/storage/itemid.h
index 60570cc..9ea9269 100644
--- a/src/include/storage/itemid.h
+++ b/src/include/storage/itemid.h
@@ -112,6 +112,11 @@ typedef uint16 ItemLength;
#define ItemIdIsDead(itemId) \
((itemId)->lp_flags == LP_DEAD)
+#define ItemIdIsDeadRedirection(itemId) \
+ ( ((itemId)->lp_flags == 3) && \
+ ((itemId)->lp_len == 0) && \
+ ((itemId)->lp_off != 0) )
+
/*
* ItemIdHasStorage
* True iff item identifier has associated storage.
@@ -155,6 +160,12 @@ typedef uint16 ItemLength;
(itemId)->lp_len = 0 \
)
+#define ItemIdSetDeadRedirect(itemId, link) \
+( \
+ (itemId)->lp_off = (link), \
+ (itemId)->lp_len = 0 \
+)
+
/*
* ItemIdSetDead
* Set the item identifier to be DEAD, with no storage.
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-18 19:59 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 20:42 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Claudio Freire <klaussfreire@gmail.com>
1 sibling, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-18 19:59 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> I have written a code for quick indextuple deletion from an relation by heap
> tuple TID. The code relate to "Retail IndexTuple deletion" enhancement of
> btree index on postgresql wiki [1].
I knew that somebody would eventually read that Wiki page. :-)
> Now, index relation cleaning performs by vacuum which scan all index
> relation for dead entries sequentially, tuple-by-tuple. This simplistic and
> safe method can be significantly surpasses in the cases, than a number of
> dead entries is not large by retail deletions which uses index scans for
> search dead entries.
I assume that the lazy vacuum bulk delete thing is much faster for the
situation where you have many dead tuples in the index. However,
allowing a B-Tree index to accumulate so much bloat in the first place
often has consequences that cannot be reversed with anything less than
a REINDEX. It's true that "prevention is better than cure" for all
types of bloat. However, this could perhaps be as much as 100x more
important for B-Tree bloat, since we cannot place new tuples in any
place that is convenient. It's very different to bloat in the heap,
even at a high level.
> The code demands hold DEAD tuple storage until scan key will be created. In
> this version I add 'target_index_deletion_factor' option. If it more than 0,
> heap_page_prune() uses ItemIdMarkDead() instead of ItemIdSetDead() function
> for set DEAD flag and hold the tuple storage.
Makes sense.
> Next step is developing background worker which will collect pairs (tid,
> scankey) of DEAD tuples from heap_page_prune() function.
Makes sense.
> | n | t1, s | t2, s | speedup |
> |---|---------------------------|
> | 0 | 0.00003| 0.4476 | 1748.4 |
> | 1 | 0.00006| 0.5367 | 855.99 |
> | 2 | 0.0004 | 0.9804 | 233.99 |
> | 3 | 0.0048 | 1.6493 | 34.576 |
> | 4 | 0.5600 | 2.4854 | 4.4382 |
> | 5 | 3.3300 | 3.9555 | 1.2012 |
> | 6 | 17.700 | 5.6000 | 0.3164 |
> |---|---------------------------|
> in the table, t1 - measured execution time of lazy_vacuum_index() function
> by Quick-Vacuum strategy; t2 - measured execution time of
> lazy_vacuum_index() function by Lazy-Vacuum strategy;
The speedup looks promising. However, the real benefit should be in
query performance, especially when we have heavy contention. Very
eager retail index tuple deletion could help a lot there. It already
makes sense to make autovacuum extremely aggressive in this case, to
the point when it's running almost constantly. A more targeted cleanup
process that can run much faster could do the same job, but be much
more eager, and so be much more effective at *preventing* bloating of
the key space [1][2].
> Note, guaranteed allowable time of index scans (used for quick deletion)
> will be achieved by storing equal-key index tuples in physical TID order [2]
> with patch [3].
I now have greater motivation to develop that patch into a real project.
I bet that my heap-tid-sort patch will allow you to refine your
interface when there are many logical duplicates: You can create one
explicit scan key, but have a list of heap TIDs that need to be killed
within the range of matching index tuples. That could be a lot more
efficient in the event of many non-HOT updates, where most index tuple
values won't actually change. You can sort the list of heap TIDs that
you want to kill once, and then "merge" it with the tuples at the leaf
level as they are matched/killed. It should be safe to avoid
rechecking anything other than the heap TID values.
[1] http://pgeoghegan.blogspot.com/2017/07/postgresql-index-bloat-microscope.html
[2] https://www.postgresql.org/message-id/CAH2-Wzmf6intNY1ggiNzOziiO5Eq=DsXfeptODGxO=2j-i1NGQ@mail.gmail...
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 19:59 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-18 20:42 ` Claudio Freire <klaussfreire@gmail.com>
2018-06-18 21:15 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 47+ messages in thread
From: Claudio Freire @ 2018-06-18 20:42 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-hackers
On Mon, Jun 18, 2018 at 4:59 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > Note, guaranteed allowable time of index scans (used for quick deletion)
> > will be achieved by storing equal-key index tuples in physical TID order [2]
> > with patch [3].
>
> I now have greater motivation to develop that patch into a real project.
>
> I bet that my heap-tid-sort patch will allow you to refine your
> interface when there are many logical duplicates: You can create one
> explicit scan key, but have a list of heap TIDs that need to be killed
> within the range of matching index tuples. That could be a lot more
> efficient in the event of many non-HOT updates, where most index tuple
> values won't actually change. You can sort the list of heap TIDs that
> you want to kill once, and then "merge" it with the tuples at the leaf
> level as they are matched/killed. It should be safe to avoid
> rechecking anything other than the heap TID values.
>
> [1] http://pgeoghegan.blogspot.com/2017/07/postgresql-index-bloat-microscope.html
> [2] https://www.postgresql.org/message-id/CAH2-Wzmf6intNY1ggiNzOziiO5Eq=DsXfeptODGxO=2j-i1NGQ@mail.gmail...
Actually, once btree tids are sorted, you can continue tree descent
all the way to the exact leaf page that contains the tuple to be
deleted.
Thus, the single-tuple interface ends up being quite OK. Sure, you can
optimize things a bit by scanning a range, but only if vacuum is able
to group keys in order to produce the optimized calls, and I don't see
that terribly likely.
So, IMHO the current interface may be quite enough.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 19:59 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 20:42 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Claudio Freire <klaussfreire@gmail.com>
@ 2018-06-18 21:15 ` Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-18 21:15 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; pgsql-hackers
On Mon, Jun 18, 2018 at 1:42 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
> Actually, once btree tids are sorted, you can continue tree descent
> all the way to the exact leaf page that contains the tuple to be
> deleted.
>
> Thus, the single-tuple interface ends up being quite OK. Sure, you can
> optimize things a bit by scanning a range, but only if vacuum is able
> to group keys in order to produce the optimized calls, and I don't see
> that terribly likely.
Andrey talked about a background worker that did processing + index
tuple deletion when handed off work by a user backend after it
performs HOT pruning of a heap page. I consider that idea to be a good
place to go with the patch, because in practice the big problem is
workloads that suffer from so-called "write amplification", where most
index tuples are created despite being "logically unnecessary" (e.g.
one index among several prevents an UPDATE being HOT-safe, making
inserts into most of the indexes "logically unnecessary").
I think that it's likely that only descending the tree once in order
to kill multiple duplicate index tuples in one shot will in fact be
*very* important (unless perhaps you assume that that problem is
solved by something else, such as zheap). The mechanism that Andrey
describes is rather unlike VACUUM as we know it today, but that's the
whole point.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-18 21:54 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-18 21:54 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> Patch '0001-retail-indextuple-deletion' introduce new function
> amtargetdelete() in access method interface. Patch
> '0002-quick-vacuum-strategy' implements this function for an alternative
> strategy of lazy index vacuum, called 'Quick Vacuum'.
My compiler shows the following warnings:
/code/postgresql/root/build/../source/src/backend/access/nbtree/nbtree.c:
In function ‘bttargetdelete’:
/code/postgresql/root/build/../source/src/backend/access/nbtree/nbtree.c:1053:3:
warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
if (needLock)
^~
/code/postgresql/root/build/../source/src/backend/access/nbtree/nbtree.c:1055:4:
note: ...this statement, but the latter is misleadingly indented as if
it were guarded by the ‘if’
npages = RelationGetNumberOfBlocks(irel);
^~~~~~
/code/postgresql/root/build/../source/src/backend/access/nbtree/nbtree.c:1074:3:
warning: ‘blkno’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
cleanup_block(info, stats, blkno);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I think that they're both harmless, though.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-18 23:05 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 02:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-20 04:04 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 4 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-18 23:05 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Mon, Jun 18, 2018 at 2:54 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> Patch '0001-retail-indextuple-deletion' introduce new function
>> amtargetdelete() in access method interface. Patch
>> '0002-quick-vacuum-strategy' implements this function for an alternative
>> strategy of lazy index vacuum, called 'Quick Vacuum'.
>
> My compiler shows the following warnings:
Some real feedback:
What we probably want to end up with here is new lazyvacuum.c code
that does processing for one heap page (and associated indexes) that
is really just a "partial" lazy vacuum. Though it won't do things like
advance relfrozenxid, it will do pruning for the heap page, index
tuple killing, and finally heap tuple killing. It will do all of these
things reliably, just like traditional lazy vacuum. This will be what
your background worker eventually uses.
I doubt that you can use routines like index_beginscan() within
bttargetdelete() at all. I think that you need something closer to
_bt_doinsert() or _bt_pagedel(), that manages its own scan (your code
should probably live in nbtpage.c). It does not make sense to teach
external, generic routines like index_beginscan() about heap TID being
an implicit final index attribute, which is one reason for this (I'm
assuming that this patch relies on my own patch). Another reason is
that you need to hold an exclusive buffer lock at the point that you
identify the tuple to be killed, until the point that you actually
kill it. You don't do that now.
IOW, the approach you've taken in bttargetdelete() isn't quite correct
because you imagine that it's okay to occasionally "lose" the index
tuple that you originally found, and just move on. That needs to be
100% reliable, or else we'll end up with index tuples that point to
the wrong heap tuples in rare cases with concurrent insertions. As I
said, we want a "partial" lazy vacuum here, which must mean that it's
reliable. Note that _bt_pagedel() actually calls _bt_search() when it
deletes a page. Your patch will not be the first patch that makes
nbtree vacuuming do an index scan. You should be managing your own
insertion scan key, much like _bt_pagedel() does. If you use my patch,
_bt_search() can be taught to look for a specific heap TID.
Finally, doing things this way would let you delete multiple
duplicates in one shot, as I described in an earlier e-mail. Only a
single descent of the tree is needed to delete quite a few index
tuples, provided that they all happen to be logical duplicates. Again,
your background worker will take advantage of this.
This code does not follow the Postgres style:
> - else
> + }
> + else {
> + if (rootoffnum != latestdead)
> + heap_prune_record_unused(prstate, latestdead);
> heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
> + }
> }
Please be more careful about that. I find it very distracting.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-18 23:33 ` Peter Geoghegan <pg@bowt.ie>
3 siblings, 0 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-18 23:33 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Mon, Jun 18, 2018 at 4:05 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> Finally, doing things this way would let you delete multiple
> duplicates in one shot, as I described in an earlier e-mail. Only a
> single descent of the tree is needed to delete quite a few index
> tuples, provided that they all happen to be logical duplicates. Again,
> your background worker will take advantage of this.
BTW, when you do this you should make sure that there is only one call
to _bt_delitems_vacuum(), so that there aren't too many WAL records.
Actually, that's not quite correct -- there should be one
_bt_delitems_vacuum() call *per leaf page* per bttargetdelete() call,
which is slightly different. There should rarely be more than one or
two calls to _bt_delitems_vacuum() in total, because your background
worker is only going to delete one heap page's duplicates per
bttargetdelete() call, and because there will be locality/correlation
with my TID order patch.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-19 02:29 ` Peter Geoghegan <pg@bowt.ie>
3 siblings, 0 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-19 02:29 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Mon, Jun 18, 2018 at 4:05 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> IOW, the approach you've taken in bttargetdelete() isn't quite correct
> because you imagine that it's okay to occasionally "lose" the index
> tuple that you originally found, and just move on. That needs to be
> 100% reliable, or else we'll end up with index tuples that point to
> the wrong heap tuples in rare cases with concurrent insertions.
Attached patch adds a new amcheck check within
bt_index_parent_check(). With the patch, heap TIDs are accumulated in
a tuplesort and sorted at the tail end of verification (before
optional heapallindexed verification runs). This will reliably detect
the kind of corruption I noticed was possible with your patch.
Note that the amcheck enhancement that went along with my
heap-tid-btree-sort patch may not have detected this issue, which is
why I wrote this patch -- the heap-tid-btree-sort amcheck stuff could
detect duplicates, but only when all other attributes happened to be
identical when comparing sibling index tuples (i.e. only when we must
actually compare TIDs across sibling index tuples). If you add this
check, I'm pretty sure that you can detect any possible problem. You
should think about using this to debug your patch.
I may get around to submitting this to a CF, but that isn't a priority
right now.
--
Peter Geoghegan
Attachments:
0001-Detect-duplicate-heap-TIDs-using-a-tuplesort.patch (application/octet-stream, 4.9K) download
From 4309a5f139289c3078f12517c31b31a8d52e5cd0 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 21 Oct 2017 17:19:55 -0700
Subject: [PATCH] Detect duplicate heap TIDs using a tuplesort.
Make bt_index_parent_check() check for duplicate TIDs. This condition
might otherwise be missed even when using bt_index_parent_check() +
healallindexed verification.
"trace_sort=on" can be used as a rudimentary progress indicator.
---
contrib/amcheck/verify_nbtree.c | 88 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 88 insertions(+)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index a1438a2855..cdd2b622f1 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -29,12 +29,15 @@
#include "access/xact.h"
#include "catalog/index.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_operator.h"
+#include "catalog/pg_type.h"
#include "commands/tablecmds.h"
#include "lib/bloomfilter.h"
#include "miscadmin.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
+#include "utils/tuplesort.h"
PG_MODULE_MAGIC;
@@ -93,6 +96,8 @@ typedef struct BtreeCheckState
bloom_filter *filter;
/* Bloom filter fingerprints downlink blocks within tree */
bloom_filter *downlinkfilter;
+ /* readonly TIDs-are-unique tuplesort */
+ Tuplesortstate *tidsort;
/* Right half of incomplete split marker */
bool rightsplit;
/* Debug counter */
@@ -144,6 +149,7 @@ static inline bool invariant_leq_nontarget_offset(BtreeCheckState *state,
Page other,
ScanKey key,
OffsetNumber upperbound);
+static inline int64 itemptr_encode(ItemPointer itemptr);
static Page palloc_btree_page(BtreeCheckState *state, BlockNumber blocknum);
/*
@@ -345,6 +351,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
state->heaprel = heaprel;
state->readonly = readonly;
state->heapallindexed = heapallindexed;
+ state->tidsort = NULL;
if (state->heapallindexed)
{
@@ -413,6 +420,21 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
}
}
+ if (state->readonly)
+ {
+ /*
+ * Encode TIDs as int8 values for the sort, rather than directly sorting
+ * item pointers. This can be significantly faster, primarily because TID
+ * is a pass-by-reference type on all platforms, whereas int8 is
+ * pass-by-value on most platforms.
+ */
+ state->tidsort = tuplesort_begin_datum(INT8OID, Int8LessOperator,
+ InvalidOid, false,
+ maintenance_work_mem, NULL,
+ false);
+ }
+
+
/* Create context for page */
state->targetcontext = AllocSetContextCreate(CurrentMemoryContext,
"amcheck context",
@@ -472,6 +494,46 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
previouslevel = current.level;
}
+ if (state->readonly)
+ {
+ /* state variables for the merge */
+ bool tuplesort_empty;
+ int64 last_tid = -1;
+
+ /* Sort TIDs to detect duplicate pointers to the heap */
+ tuplesort_performsort(state->tidsort);
+
+ while (!tuplesort_empty)
+ {
+ Datum ts_val;
+ bool ts_isnull;
+ int64 ts_tid;
+
+ tuplesort_empty = !tuplesort_getdatum(state->tidsort, true,
+ &ts_val, &ts_isnull, NULL);
+ Assert(tuplesort_empty || !ts_isnull);
+ if (!tuplesort_empty)
+ {
+ ts_tid = DatumGetInt64(ts_val);
+
+ if (last_tid >= ts_tid)
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("duplicate heap TID")));
+
+ last_tid = ts_tid;
+
+ /* If int8 is pass-by-ref, free (encoded) TID Datum memory */
+#ifndef USE_FLOAT8_BYVAL
+ pfree(DatumGetPointer(ts_val));
+#endif
+ }
+ }
+
+ /* Done with tuplesort object */
+ tuplesort_end(state->tidsort);
+ }
+
/*
* * Check whether heap contains unindexed/malformed tuples *
*/
@@ -909,6 +971,14 @@ bt_target_page_check(BtreeCheckState *state)
if (state->heapallindexed && P_ISLEAF(topaque) && !ItemIdIsDead(itemid))
bloom_add_element(state->filter, (unsigned char *) itup, tupsize);
+ /* When checking for duplicate TIDs, add heap TIDs here */
+ if (state->tidsort && P_ISLEAF(topaque))
+ {
+ int64 encoded = itemptr_encode(&itup->t_tid);
+
+ tuplesort_putdatum(state->tidsort, Int64GetDatum(encoded), false);
+ }
+
/*
* * High key check *
*
@@ -1812,6 +1882,24 @@ invariant_leq_nontarget_offset(BtreeCheckState *state,
return cmp <= 0;
}
+/*
+ * itemptr_encode - Encode ItemPointer as int64/int8
+ *
+ * This is lifted from index.c. This is a performance optimization for sorting
+ * TIDs.
+ */
+static inline int64
+itemptr_encode(ItemPointer itemptr)
+{
+ BlockNumber block = ItemPointerGetBlockNumber(itemptr);
+ OffsetNumber offset = ItemPointerGetOffsetNumber(itemptr);
+ int64 encoded;
+
+ encoded = ((uint64) block << 16) | (uint16) offset;
+
+ return encoded;
+}
+
/*
* Given a block number of a B-Tree page, return page in palloc()'d memory.
* While at it, perform some basic checks of the page.
--
2.14.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-19 09:33 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
3 siblings, 1 reply; 47+ messages in thread
From: Masahiko Sawada @ 2018-06-19 09:33 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; pgsql-hackers
On Tue, Jun 19, 2018 at 8:05 AM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Mon, Jun 18, 2018 at 2:54 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov
>> <a.lepikhov@postgrespro.ru> wrote:
>>> Patch '0001-retail-indextuple-deletion' introduce new function
>>> amtargetdelete() in access method interface. Patch
>>> '0002-quick-vacuum-strategy' implements this function for an alternative
>>> strategy of lazy index vacuum, called 'Quick Vacuum'.
Great!
>>
>> My compiler shows the following warnings:
>
> Some real feedback:
>
> What we probably want to end up with here is new lazyvacuum.c code
> that does processing for one heap page (and associated indexes) that
> is really just a "partial" lazy vacuum. Though it won't do things like
> advance relfrozenxid, it will do pruning for the heap page, index
> tuple killing, and finally heap tuple killing. It will do all of these
> things reliably, just like traditional lazy vacuum. This will be what
> your background worker eventually uses.
I think that we do the partial lazy vacuum using visibility map even
now. That does heap pruning, index tuple killing but doesn't advance
relfrozenxid. Since this patch adds an ability to delete small amount
of index tuples quickly, what I'd like to do with this patch is to
invoke autovacuum more frequently, and do the target index deletion or
the index bulk-deletion depending on amount of garbage and index size
etc. That is, it might be better if lazy vacuum scans heap in ordinary
way and then plans and decides a method of index deletion based on
costs similar to what query planning does.
Regards,
--
Masahiko Sawada
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-06-19 17:38 ` Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-19 17:38 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; pgsql-hackers
On Tue, Jun 19, 2018 at 2:33 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> I think that we do the partial lazy vacuum using visibility map even
> now. That does heap pruning, index tuple killing but doesn't advance
> relfrozenxid.
Right, that's what I was thinking. Opportunistic HOT pruning isn't
like vacuuming because it doesn't touch indexes. This patch adds an
alternative strategy for conventional lazy vacuum that is also able to
run a page at a time if needed. Perhaps page-at-a-time operation could
later be used for doing something that is opportunistic in the same
way that pruning is opportunistic, but it's too early to worry about
that.
> Since this patch adds an ability to delete small amount
> of index tuples quickly, what I'd like to do with this patch is to
> invoke autovacuum more frequently, and do the target index deletion or
> the index bulk-deletion depending on amount of garbage and index size
> etc. That is, it might be better if lazy vacuum scans heap in ordinary
> way and then plans and decides a method of index deletion based on
> costs similar to what query planning does.
That seems to be what Andrey wants to do, though right now the
prototype patch actually just always uses its alternative strategy
while doing any kind of lazy vacuuming (some simple costing code is
commented out right now). It shouldn't be too hard to add some costing
to it. Once we do that, and once we polish the patch some more, we can
do performance testing. Maybe that alone will be enough to make the
patch worth committing; "opportunistic microvacuuming" can come later,
if at all.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-22 11:24 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-26 10:31 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-22 11:24 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-hackers
Hi,
According to your feedback, i develop second version of the patch.
In this version:
1. High-level functions index_beginscan(), index_rescan() not used. Tree
descent made by _bt_search(). _bt_binsrch() used for positioning on the
page.
2. TID list introduced in amtargetdelete() interface. Now only one tree
descent needed for deletion all tid's from the list with equal scan key
value - logical duplicates deletion problem.
3. Only one WAL record for index tuple deletion per leaf page per
amtargetdelete() call.
4. VACUUM can sort TID list preliminary for more quick search of duplicates.
Background worker will come later.
On 19.06.2018 22:38, Peter Geoghegan wrote:
> On Tue, Jun 19, 2018 at 2:33 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> I think that we do the partial lazy vacuum using visibility map even
>> now. That does heap pruning, index tuple killing but doesn't advance
>> relfrozenxid.
>
> Right, that's what I was thinking. Opportunistic HOT pruning isn't
> like vacuuming because it doesn't touch indexes. This patch adds an
> alternative strategy for conventional lazy vacuum that is also able to
> run a page at a time if needed. Perhaps page-at-a-time operation could
> later be used for doing something that is opportunistic in the same
> way that pruning is opportunistic, but it's too early to worry about
> that.
>
>> Since this patch adds an ability to delete small amount
>> of index tuples quickly, what I'd like to do with this patch is to
>> invoke autovacuum more frequently, and do the target index deletion or
>> the index bulk-deletion depending on amount of garbage and index size
>> etc. That is, it might be better if lazy vacuum scans heap in ordinary
>> way and then plans and decides a method of index deletion based on
>> costs similar to what query planning does.
>
> That seems to be what Andrey wants to do, though right now the
> prototype patch actually just always uses its alternative strategy
> while doing any kind of lazy vacuuming (some simple costing code is
> commented out right now). It shouldn't be too hard to add some costing
> to it. Once we do that, and once we polish the patch some more, we can
> do performance testing. Maybe that alone will be enough to make the
> patch worth committing; "opportunistic microvacuuming" can come later,
> if at all.
>
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
Attachments: download all (2 files, .tar.gz)
0001-retail-indextuple-deletion.patch (text/x-patch, 11.8K) download
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3..96f1d47 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbc..a0e06bd 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182..acf14e7 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42eff..d7a53d2 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df3..5fb32d6 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 22b5cc9..9ebeb78 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -730,6 +730,21 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexTargetDeleteResult *
+index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation indexRelation = info->indexRelation;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, values, isnull);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 27a3032..b6f01a0 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -34,7 +34,9 @@
#include "storage/smgr.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/syscache.h"
/* Working state needed by btvacuumpage */
@@ -127,6 +129,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -884,6 +887,173 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+static int
+tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid, bool IsSorted)
+{
+ if (!IsSorted)
+ {
+ for (int i=0; i< ntid; i++)
+ if (ItemPointerEquals(tid, &(tid_list[i])))
+ return i;
+ }
+ else
+ {
+ int low = 0,
+ high = ntid,
+ mid;
+ /* Search at sorted list of TID*/
+ if ((ItemPointerCompare(tid, &tid_list[low]) >= 0) && (ItemPointerCompare(tid, &tid_list[high]) <= 0))
+ {
+ for (;;)
+ {
+ mid = (low+high)/2;
+
+ if ((mid == low) || (mid == high))
+ break;
+ if (ItemPointerCompare(tid, &tid_list[mid]) < 0)
+ if (high == mid)
+ break;
+ else
+ high = mid;
+ else if (ItemPointerCompare(tid, &tid_list[mid]) > 0)
+ if (low == mid)
+ break;
+ else
+ low = mid;
+ else
+ return mid;
+ }
+ }
+ }
+ return -1;
+}
+
+IndexTargetDeleteResult*
+bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation irel = info->indexRelation;
+ Relation hrel = info->heapRelation;
+ ScanKey skey;
+ int keysCount = IndexRelationGetNumberOfKeyAttributes(irel);
+ BTStack stack;
+ Buffer buf;
+ Page page;
+ BTPageOpaque opaque;
+ OffsetNumber offnum;
+ int ndeletable = 0;
+ OffsetNumber deletable[MaxOffsetNumber];
+ IndexTuple itup;
+
+ if (stats == NULL)
+ stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+
+ itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
+ skey = _bt_mkscankey(irel, itup);
+
+ /* Descend the tree and position ourselves on the target leaf page. */
+ stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
+ _bt_freestack(stack);
+
+ /* To prepare tuple entries search across index pages */
+ Assert(BufferIsValid(buf));
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ for (;;)
+ {
+ int32 cmpval;
+ ItemId itemid;
+ IndexTuple itup;
+ int pos;
+
+ /* Switch to the next page */
+ if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ {
+ /*
+ * Before unlocking index page we need to delete
+ * all currently found tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ ndeletable = 0;
+ }
+
+ /*
+ * Check for end-of-index
+ */
+ if (P_RIGHTMOST(opaque))
+ /* it is rightmost leaf */
+ break;
+
+ /*
+ * Switch to the next index page
+ */
+ buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = P_FIRSTDATAKEY(opaque);
+ continue;
+ }
+
+ /*
+ * This index entry satisfied to the scan key?
+ */
+ cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+
+ if (cmpval != 0)
+ /* End of index entries, satisfied to the scan key */
+ break;
+
+ /*
+ * To load index tuple and look for matches in the TID list of
+ * dead heap tuples
+ */
+ itemid = PageGetItemId(page, offnum);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples, false);
+
+ if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+
+ offnum = OffsetNumberNext(offnum);
+ }
+
+ /*
+ * Delete all found index tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ }
+
+ /* Release scan key, unpin and unlock buffer */
+ _bt_freeskey(skey);
+ _bt_relbuf(irel, buf);
+
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4a9b5da..3257281 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -56,6 +56,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6..497b54e 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,11 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexTargetDeleteResult *(*amtargetdelete_function) (IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 24c720b..90b7e6f 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -51,6 +51,16 @@ typedef struct IndexVacuumInfo
BufferAccessStrategy strategy; /* access strategy for reads */
} IndexVacuumInfo;
+typedef struct IndexTargetDeleteInfo
+{
+ Relation heapRelation;
+ Relation indexRelation; /* the index being vacuumed */
+ int num_dead_tuples;
+ bool isSorted; /* dead tuples list sorted */
+ ItemPointer dead_tuples;
+ bool* found_dead_tuples;
+} IndexTargetDeleteInfo;
+
/*
* Struct for statistics returned by ambulkdelete and amvacuumcleanup
*
@@ -79,6 +89,11 @@ typedef struct IndexBulkDeleteResult
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+typedef struct IndexTargetDeleteResult
+{
+ int tuples_removed; /* # removed during vacuum operation */
+} IndexTargetDeleteResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
@@ -163,6 +178,10 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexTargetDeleteResult *index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 04ecb4c..3b9461c 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -508,6 +508,10 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexTargetDeleteResult *bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
0002-quick-vacuum-strategy.patch (text/x-patch, 10.7K) download
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343..06efc90 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -405,7 +408,7 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (HeapTupleSatisfiesVacuum(&tup, OldestXmin, buffer)
== HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
{
- heap_prune_record_unused(prstate, rootoffnum);
+ heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
ndeleted++;
@@ -580,8 +583,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
ndeleted++;
+ if (chainitems[i] == latestdead)
+ continue;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +603,25 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ ItemIdSetDeadRedirect(rootlp, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
+ }
else
+ {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -706,7 +727,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 5649a70..3847104 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -139,7 +139,6 @@ typedef struct LVRelStats
bool lock_waiter_detected;
} LVRelStats;
-
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
@@ -156,6 +155,9 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
+static void quick_vacuum_index(Relation irel, Relation hrel,
+ IndexBulkDeleteResult **stats,
+ LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -734,10 +736,17 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy)
+ quick_vacuum_index(Irel[i], onerel, &indstats[i], vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
+ }
/*
* Report that we are now vacuuming the heap. We also increase
* the number of index scans here; note that by using
@@ -1379,10 +1388,16 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy)
+ quick_vacuum_index(Irel[i], onerel, &indstats[i], vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
-
+ }
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
@@ -1671,6 +1686,149 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Get tuple from heap for a scan key building.
+ */
+static HeapTuple
+get_tuple_by_tid(Relation rel, ItemPointer tid)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ ItemId lp;
+ HeapTuple tuple;
+
+ buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
+ LockBuffer(buffer, BUFFER_LOCK_SHARE);
+
+ page = (Page) BufferGetPage(buffer);
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
+
+ /*
+ * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
+ */
+ if (!ItemIdIsUsed(lp))
+ return NULL;
+
+ /* Walk along the chain */
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+ Assert(ItemIdIsUsed(lp));
+ }
+
+ /* Form a tuple */
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(rel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ UnlockReleaseBuffer(buffer);
+ return tuple;
+}
+
+#include "access/nbtree.h"
+#include "catalog/index.h"
+#include "executor/executor.h"
+
+static int tid_comparator(const void* a, const void* b)
+{
+ return ItemPointerCompare((ItemPointer)a, (ItemPointer)b);
+}
+
+/*
+ * quick_vacuum_index() -- quick vacuum one index relation.
+ *
+ * Delete all the index entries pointing to tuples listed in
+ * vacrelstats->dead_tuples.
+ */
+static void
+quick_vacuum_index(Relation irel, Relation hrel,
+ IndexBulkDeleteResult **overall_stats,
+ LVRelStats *vacrelstats)
+{
+ IndexTargetDeleteResult stats;
+ IndexTargetDeleteInfo ivinfo;
+
+ if (irel->rd_amroutine->amtargetdelete != NULL)
+ {
+ int tnum;
+ bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
+ IndexInfo* indexInfo = BuildIndexInfo(irel);
+ EState* estate = CreateExecutorState();
+ ExprContext* econtext = GetPerTupleExprContext(estate);
+ ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+ qsort((void *)vacrelstats->dead_tuples, vacrelstats->num_dead_tuples, sizeof(ItemPointerData), tid_comparator);
+ ivinfo.isSorted = true;
+
+ /* Get tuple from heap */
+ for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ {
+ HeapTuple tuple;
+ TupleTableSlot* slot;
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+
+ /* Index entry for the TID was deleted early */
+ if (found[tnum])
+ continue;
+
+ if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ {
+ /*
+ * Tuple has 'not used' status.
+ */
+ found[tnum] = true;
+ continue;
+ }
+
+ /*
+ * Form values[] and isnull[] arrays from for index tuple
+ * by heap tuple
+ */
+ slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ econtext->ecxt_scantuple = slot;
+
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * In a partial index, ignore tuples that don't satisfy the
+ * predicate.
+ */
+ if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
+ {
+ found[tnum] = true;
+ continue;
+ }
+
+ FormIndexDatum(indexInfo, slot, estate, values, isnull);
+
+ ExecDropSingleTupleTableSlot(slot);
+
+ /*
+ * Make attempt to delete some index entries by one tree descent.
+ * We use only a part of TID list, which contains not found TID's.
+ */
+ ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
+ ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples-tnum;
+ ivinfo.found_dead_tuples = found+tnum;
+ index_target_delete(&ivinfo, &stats, values, isnull);
+ }
+
+ pfree(found);
+ FreeExecutorState(estate);
+ }
+
+ /*
+ * Collect statistical info
+ */
+ lazy_cleanup_index(irel, *overall_stats, vacrelstats);
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
@@ -1698,7 +1856,7 @@ lazy_vacuum_index(Relation indrel,
/* Do bulk deletion */
*stats = index_bulk_delete(&ivinfo, *stats,
- lazy_tid_reaped, (void *) vacrelstats);
+ lazy_tid_reaped, (void *) vacrelstats);
ereport(elevel,
(errmsg("scanned index \"%s\" to remove %d row versions",
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fa3c8a7..3f6c83b 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3236,7 +3236,15 @@ static struct config_real ConfigureNamesReal[] =
0.1, 0.0, 100.0,
NULL, NULL, NULL
},
-
+ {
+ {"target_index_deletion_factor", PGC_SIGHUP, AUTOVACUUM,
+ gettext_noop("Maximum number of vacuumed tuples as a fraction of reltuples where we can use target index vacuum strategy."),
+ NULL
+ },
+ &target_index_deletion_factor,
+ 0.01, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
{
{"checkpoint_completion_target", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval."),
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7..c5a7bc8 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,7 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
/*
* HeapScanIsValid
* True iff the heap scan is valid.
diff --git a/src/include/storage/itemid.h b/src/include/storage/itemid.h
index 60570cc..9ea9269 100644
--- a/src/include/storage/itemid.h
+++ b/src/include/storage/itemid.h
@@ -112,6 +112,11 @@ typedef uint16 ItemLength;
#define ItemIdIsDead(itemId) \
((itemId)->lp_flags == LP_DEAD)
+#define ItemIdIsDeadRedirection(itemId) \
+ ( ((itemId)->lp_flags == 3) && \
+ ((itemId)->lp_len == 0) && \
+ ((itemId)->lp_off != 0) )
+
/*
* ItemIdHasStorage
* True iff item identifier has associated storage.
@@ -155,6 +160,12 @@ typedef uint16 ItemLength;
(itemId)->lp_len = 0 \
)
+#define ItemIdSetDeadRedirect(itemId, link) \
+( \
+ (itemId)->lp_off = (link), \
+ (itemId)->lp_len = 0 \
+)
+
/*
* ItemIdSetDead
* Set the item identifier to be DEAD, with no storage.
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-22 19:43 ` Peter Geoghegan <pg@bowt.ie>
2018-06-22 20:14 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
1 sibling, 2 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-22 19:43 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> According to your feedback, i develop second version of the patch.
> In this version:
> 1. High-level functions index_beginscan(), index_rescan() not used. Tree
> descent made by _bt_search(). _bt_binsrch() used for positioning on the
> page.
> 2. TID list introduced in amtargetdelete() interface. Now only one tree
> descent needed for deletion all tid's from the list with equal scan key
> value - logical duplicates deletion problem.
> 3. Only one WAL record for index tuple deletion per leaf page per
> amtargetdelete() call.
Cool.
What is this "race" code about?
> + buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
> + LockBuffer(buffer, BUFFER_LOCK_SHARE);
> +
> + page = (Page) BufferGetPage(buffer);
> + offnum = ItemPointerGetOffsetNumber(tid);
> + lp = PageGetItemId(page, offnum);
> +
> + /*
> + * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
> + */
> + if (!ItemIdIsUsed(lp))
> + return NULL;
It looks wrong -- why should another process have set the item as
unused? And if we assume that that's possible at all, what's to stop a
third process from actually reusing the item pointer before we arrive
(at get_tuple_by_tid()), leaving us to find a tuple that is totally
unrelated to the original tuple to be deleted?
(Also, you're not releasing the buffer lock before you return.)
> 4. VACUUM can sort TID list preliminary for more quick search of duplicates.
This version of the patch prevents my own "force unique keys" patch
from working, since you're not using my proposed new
_bt_search()/_bt_binsrch()/_bt_compare() interface (you're not passing
them a heap TID). It is essential that your patch be able to quickly
reach any tuple that it needs to kill. Otherwise, the worst case
performance is totally unacceptable; it will never be okay to go
through 10%+ of the index to kill a single tuple hundreds or even
thousands of times per VACUUM. It seems to me that doing this
tid_list_search() binary search is pointless -- you should instead be
relying on logical duplicates using their heap TID as a tie-breaker.
Rather than doing a binary search within tid_list_search(), you should
instead match the presorted heap TIDs at the leaf level against the
sorted in-memory TID list. You know, a bit like a merge join.
I suggest that you go even further than this: I think that you should
just start distributing my patch as part of your patch series. You can
change my code if you need to. I also suggest using "git format patch"
with simple, short commit messages to produce patches. This makes it a
lot easier to track the version of the patch, changes over time, etc.
I understand why you'd hesitate to take ownership of my code (it has
big problems!), but the reality is that all the problems that my patch
has are also problems for your patch. One patch cannot get committed
without the other, so they are already the same project. As a bonus,
my patch will probably improve the best case performance for your
patch, since multi-deletions will now have much better locality of
access.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-22 20:14 ` Peter Geoghegan <pg@bowt.ie>
2018-06-25 18:20 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
1 sibling, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-22 20:14 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Fri, Jun 22, 2018 at 12:43 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> According to your feedback, i develop second version of the patch.
>> In this version:
>> 1. High-level functions index_beginscan(), index_rescan() not used. Tree
>> descent made by _bt_search(). _bt_binsrch() used for positioning on the
>> page.
>> 2. TID list introduced in amtargetdelete() interface. Now only one tree
>> descent needed for deletion all tid's from the list with equal scan key
>> value - logical duplicates deletion problem.
>> 3. Only one WAL record for index tuple deletion per leaf page per
>> amtargetdelete() call.
>
> Cool.
>
> What is this "race" code about?
I noticed another bug in your patch, when running a
"wal_consistency_checking=all" smoke test. I do this simple, generic
test for anything that touches WAL-logging, actually -- it's a good
practice to adopt.
I enable "wal_consistency_checking=all" on the installation, create a
streaming replica with pg_basebackup (which also has
"wal_consistency_checking=all"), and then run "make installcheck"
against the primary. Here is what I see on the standby when I do this
with v2 of your patch applied:
9524/2018-06-22 13:03:12 PDT LOG: entering standby mode
9524/2018-06-22 13:03:12 PDT LOG: consistent recovery state reached
at 0/30000D0
9524/2018-06-22 13:03:12 PDT LOG: invalid record length at 0/30000D0:
wanted 24, got 0
9523/2018-06-22 13:03:12 PDT LOG: database system is ready to accept
read only connections
9528/2018-06-22 13:03:12 PDT LOG: started streaming WAL from primary
at 0/3000000 on timeline 1
9524/2018-06-22 13:03:12 PDT LOG: redo starts at 0/30000D0
9524/2018-06-22 13:03:32 PDT FATAL: inconsistent page found, rel
1663/16384/1259, forknum 0, blkno 0
9524/2018-06-22 13:03:32 PDT CONTEXT: WAL redo at 0/3360B00 for
Heap2/CLEAN: remxid 599
9523/2018-06-22 13:03:32 PDT LOG: startup process (PID 9524) exited
with exit code 1
9523/2018-06-22 13:03:32 PDT LOG: terminating any other active server processes
9523/2018-06-22 13:03:32 PDT LOG: database system is shut down
I haven't investigated this at all, but I assume that the problem is a
simple oversight. The new ItemIdSetDeadRedirect() concept that you've
introduced probably necessitates changes in both the WAL logging
routines and the redo/recovery routines. You need to go make those
changes. (By the way, I don't think you should be using the constant
"3" with the ItemIdIsDeadRedirection() macro definition.)
Let me know if you get stuck on this, or need more direction.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 20:14 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-25 18:20 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 0 replies; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-25 18:20 UTC (permalink / raw)
To: pgsql-hackers
On 23.06.2018 01:14, Peter Geoghegan wrote:
> On Fri, Jun 22, 2018 at 12:43 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov
>> <a.lepikhov@postgrespro.ru> wrote:
>>> According to your feedback, i develop second version of the patch.
>>> In this version:
>>> 1. High-level functions index_beginscan(), index_rescan() not used. Tree
>>> descent made by _bt_search(). _bt_binsrch() used for positioning on the
>>> page.
>>> 2. TID list introduced in amtargetdelete() interface. Now only one tree
>>> descent needed for deletion all tid's from the list with equal scan key
>>> value - logical duplicates deletion problem.
>>> 3. Only one WAL record for index tuple deletion per leaf page per
>>> amtargetdelete() call.
>>
>> Cool.
>>
>> What is this "race" code about?
>
I introduce this check because keep in mind about another vacuum
workers, which can make cleaning a relation concurrently. may be it is
redundant.
> I noticed another bug in your patch, when running a
> "wal_consistency_checking=all" smoke test. I do this simple, generic
> test for anything that touches WAL-logging, actually -- it's a good
> practice to adopt.
>
> I enable "wal_consistency_checking=all" on the installation, create a
> streaming replica with pg_basebackup (which also has
> "wal_consistency_checking=all"), and then run "make installcheck"
> against the primary. Here is what I see on the standby when I do this
> with v2 of your patch applied:
>
> 9524/2018-06-22 13:03:12 PDT LOG: entering standby mode
> 9524/2018-06-22 13:03:12 PDT LOG: consistent recovery state reached
> at 0/30000D0
> 9524/2018-06-22 13:03:12 PDT LOG: invalid record length at 0/30000D0:
> wanted 24, got 0
> 9523/2018-06-22 13:03:12 PDT LOG: database system is ready to accept
> read only connections
> 9528/2018-06-22 13:03:12 PDT LOG: started streaming WAL from primary
> at 0/3000000 on timeline 1
> 9524/2018-06-22 13:03:12 PDT LOG: redo starts at 0/30000D0
> 9524/2018-06-22 13:03:32 PDT FATAL: inconsistent page found, rel
> 1663/16384/1259, forknum 0, blkno 0
> 9524/2018-06-22 13:03:32 PDT CONTEXT: WAL redo at 0/3360B00 for
> Heap2/CLEAN: remxid 599
> 9523/2018-06-22 13:03:32 PDT LOG: startup process (PID 9524) exited
> with exit code 1
> 9523/2018-06-22 13:03:32 PDT LOG: terminating any other active server processes
> 9523/2018-06-22 13:03:32 PDT LOG: database system is shut down
>
> I haven't investigated this at all, but I assume that the problem is a
> simple oversight. The new ItemIdSetDeadRedirect() concept that you've
> introduced probably necessitates changes in both the WAL logging
> routines and the redo/recovery routines. You need to go make those
> changes. (By the way, I don't think you should be using the constant
> "3" with the ItemIdIsDeadRedirection() macro definition.)
>
> Let me know if you get stuck on this, or need more direction.
>
I was investigated the bug of the simple smoke test. You're right: make
any manipulations with line pointer in heap_page_prune() without
reflection in WAL record is no good idea.
But this consistency problem arises even on clean PostgreSQL
installation (without my patch) with ItemIdSetDead() -> ItemIdMarkDead()
replacement.
Byte-by-byte comparison of master and replay pages shows only 2 bytes
difference in the tuple storage part of page.
I don't stuck on yet, but good ideas are welcome.
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-27 06:40 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-29 05:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2018-06-29 09:03 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 3 replies; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-27 06:40 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On 23.06.2018 00:43, Peter Geoghegan wrote:
> On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> According to your feedback, i develop second version of the patch.
>> In this version:
>> 1. High-level functions index_beginscan(), index_rescan() not used. Tree
>> descent made by _bt_search(). _bt_binsrch() used for positioning on the
>> page.
>> 2. TID list introduced in amtargetdelete() interface. Now only one tree
>> descent needed for deletion all tid's from the list with equal scan key
>> value - logical duplicates deletion problem.
>> 3. Only one WAL record for index tuple deletion per leaf page per
>> amtargetdelete() call.
>
> Cool.
>
> What is this "race" code about?
>
>> + buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
>> + LockBuffer(buffer, BUFFER_LOCK_SHARE);
>> +
>> + page = (Page) BufferGetPage(buffer);
>> + offnum = ItemPointerGetOffsetNumber(tid);
>> + lp = PageGetItemId(page, offnum);
>> +
>> + /*
>> + * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
>> + */
>> + if (!ItemIdIsUsed(lp))
>> + return NULL;
>
> It looks wrong -- why should another process have set the item as
> unused? And if we assume that that's possible at all, what's to stop a
> third process from actually reusing the item pointer before we arrive
> (at get_tuple_by_tid()), leaving us to find a tuple that is totally
> unrelated to the original tuple to be deleted?
>
> (Also, you're not releasing the buffer lock before you return.)
>
>> 4. VACUUM can sort TID list preliminary for more quick search of duplicates.
>
> This version of the patch prevents my own "force unique keys" patch
> from working, since you're not using my proposed new
> _bt_search()/_bt_binsrch()/_bt_compare() interface (you're not passing
> them a heap TID). It is essential that your patch be able to quickly
> reach any tuple that it needs to kill. Otherwise, the worst case
> performance is totally unacceptable; it will never be okay to go
> through 10%+ of the index to kill a single tuple hundreds or even
> thousands of times per VACUUM. It seems to me that doing this
> tid_list_search() binary search is pointless -- you should instead be
> relying on logical duplicates using their heap TID as a tie-breaker.
> Rather than doing a binary search within tid_list_search(), you should
> instead match the presorted heap TIDs at the leaf level against the
> sorted in-memory TID list. You know, a bit like a merge join.
I agree with you. Binary search was developed in awaiting your patch.
>
> I suggest that you go even further than this: I think that you should
> just start distributing my patch as part of your patch series. You can
> change my code if you need to.
Good. I am ready to start distributing your patch. At the beginning of
the work I planned to make patch for physical TID ordering in the btree
index. Your patch will make it much easier.
I also suggest using "git format patch"
> with simple, short commit messages to produce patches. This makes it a
> lot easier to track the version of the patch, changes over time, etc.
>
Ok
> I understand why you'd hesitate to take ownership of my code (it has
> big problems!), but the reality is that all the problems that my patch
> has are also problems for your patch. One patch cannot get committed
> without the other, so they are already the same project. As a bonus,
> my patch will probably improve the best case performance for your
> patch, since multi-deletions will now have much better locality of
> access.
>
I still believe that the patch for physical TID ordering in btree:
1) has its own value, not only for target deletion,
2) will require only a few local changes in my code,
and this patches can be developed independently.
I prepare third version of the patches. Summary:
1. Mask DEAD tuples at a page during consistency checking (See comments
for the mask_dead_tuples() function).
2. Still not using physical TID ordering.
3. Index cleanup() after each quick_vacuum_index() call was excluded.
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
Attachments: download all (2 files, .tar.gz)
0001-Retail-IndexTuple-Deletion-patch-v.3.patch (text/x-patch, 12.6K) download
From b3f8ddf54c5bd68b96db8cda4f5f382b86a4568e Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 27 Jun 2018 09:57:50 +0500
Subject: [PATCH 1/2] Retail IndexTuple Deletion patch v.3
---
contrib/bloom/blutils.c | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 15 ++++
src/backend/access/nbtree/nbtree.c | 170 +++++++++++++++++++++++++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/include/access/amapi.h | 6 ++
src/include/access/genam.h | 19 ++++
src/include/access/nbtree.h | 4 +
11 files changed, 220 insertions(+)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3..96f1d47 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbc..a0e06bd 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182..acf14e7 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42eff..d7a53d2 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df3..5fb32d6 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 22b5cc9..9ebeb78 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -730,6 +730,21 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexTargetDeleteResult *
+index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation indexRelation = info->indexRelation;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, values, isnull);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 27a3032..b6f01a0 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -34,7 +34,9 @@
#include "storage/smgr.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/syscache.h"
/* Working state needed by btvacuumpage */
@@ -127,6 +129,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -884,6 +887,173 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+static int
+tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid, bool IsSorted)
+{
+ if (!IsSorted)
+ {
+ for (int i=0; i< ntid; i++)
+ if (ItemPointerEquals(tid, &(tid_list[i])))
+ return i;
+ }
+ else
+ {
+ int low = 0,
+ high = ntid,
+ mid;
+ /* Search at sorted list of TID*/
+ if ((ItemPointerCompare(tid, &tid_list[low]) >= 0) && (ItemPointerCompare(tid, &tid_list[high]) <= 0))
+ {
+ for (;;)
+ {
+ mid = (low+high)/2;
+
+ if ((mid == low) || (mid == high))
+ break;
+ if (ItemPointerCompare(tid, &tid_list[mid]) < 0)
+ if (high == mid)
+ break;
+ else
+ high = mid;
+ else if (ItemPointerCompare(tid, &tid_list[mid]) > 0)
+ if (low == mid)
+ break;
+ else
+ low = mid;
+ else
+ return mid;
+ }
+ }
+ }
+ return -1;
+}
+
+IndexTargetDeleteResult*
+bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation irel = info->indexRelation;
+ Relation hrel = info->heapRelation;
+ ScanKey skey;
+ int keysCount = IndexRelationGetNumberOfKeyAttributes(irel);
+ BTStack stack;
+ Buffer buf;
+ Page page;
+ BTPageOpaque opaque;
+ OffsetNumber offnum;
+ int ndeletable = 0;
+ OffsetNumber deletable[MaxOffsetNumber];
+ IndexTuple itup;
+
+ if (stats == NULL)
+ stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+
+ itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
+ skey = _bt_mkscankey(irel, itup);
+
+ /* Descend the tree and position ourselves on the target leaf page. */
+ stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
+ _bt_freestack(stack);
+
+ /* To prepare tuple entries search across index pages */
+ Assert(BufferIsValid(buf));
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ for (;;)
+ {
+ int32 cmpval;
+ ItemId itemid;
+ IndexTuple itup;
+ int pos;
+
+ /* Switch to the next page */
+ if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ {
+ /*
+ * Before unlocking index page we need to delete
+ * all currently found tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ ndeletable = 0;
+ }
+
+ /*
+ * Check for end-of-index
+ */
+ if (P_RIGHTMOST(opaque))
+ /* it is rightmost leaf */
+ break;
+
+ /*
+ * Switch to the next index page
+ */
+ buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = P_FIRSTDATAKEY(opaque);
+ continue;
+ }
+
+ /*
+ * This index entry satisfied to the scan key?
+ */
+ cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+
+ if (cmpval != 0)
+ /* End of index entries, satisfied to the scan key */
+ break;
+
+ /*
+ * To load index tuple and look for matches in the TID list of
+ * dead heap tuples
+ */
+ itemid = PageGetItemId(page, offnum);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples, false);
+
+ if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+
+ offnum = OffsetNumberNext(offnum);
+ }
+
+ /*
+ * Delete all found index tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ }
+
+ /* Release scan key, unpin and unlock buffer */
+ _bt_freeskey(skey);
+ _bt_relbuf(irel, buf);
+
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4a9b5da..3257281 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -56,6 +56,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6..497b54e 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,11 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexTargetDeleteResult *(*amtargetdelete_function) (IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 24c720b..90b7e6f 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -51,6 +51,16 @@ typedef struct IndexVacuumInfo
BufferAccessStrategy strategy; /* access strategy for reads */
} IndexVacuumInfo;
+typedef struct IndexTargetDeleteInfo
+{
+ Relation heapRelation;
+ Relation indexRelation; /* the index being vacuumed */
+ int num_dead_tuples;
+ bool isSorted; /* dead tuples list sorted */
+ ItemPointer dead_tuples;
+ bool* found_dead_tuples;
+} IndexTargetDeleteInfo;
+
/*
* Struct for statistics returned by ambulkdelete and amvacuumcleanup
*
@@ -79,6 +89,11 @@ typedef struct IndexBulkDeleteResult
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+typedef struct IndexTargetDeleteResult
+{
+ int tuples_removed; /* # removed during vacuum operation */
+} IndexTargetDeleteResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
@@ -163,6 +178,10 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexTargetDeleteResult *index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 04ecb4c..3b9461c 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -508,6 +508,10 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexTargetDeleteResult *bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
--
2.7.4
0002-Quick-Vacuum-Strategy-patch-v.3.patch (text/x-patch, 11.9K) download
From b33cd81ba0e92e02077d133009ac2ac9addc0983 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 27 Jun 2018 10:14:19 +0500
Subject: [PATCH 2/2] Quick Vacuum Strategy patch v.3
---
src/backend/access/heap/heapam.c | 31 +++++++
src/backend/access/heap/pruneheap.c | 40 ++++++++-
src/backend/commands/vacuumlazy.c | 162 +++++++++++++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 10 ++-
src/include/access/heapam.h | 1 +
5 files changed, 235 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 72395a5..bf964f8 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -9393,6 +9393,36 @@ heap_sync(Relation rel)
}
/*
+ * Mask DEAD tuples at a HEAP page
+ *
+ * We introduce this function for wal_consistency_checking satisfaction at
+ * Hot Standby node.
+ *
+ * DEAD tuple on a master node has t_cid and t_infomask, which can not be
+ * obtained by WAL records applying on a Hot Standby node.
+ */
+static void
+mask_dead_tuples(Page page)
+{
+ OffsetNumber offnum;
+
+ for (offnum = FirstOffsetNumber; offnum <= PageGetMaxOffsetNumber(page); offnum = OffsetNumberNext(offnum))
+ {
+ ItemId lp = PageGetItemId(page, offnum);
+ char* htup_begin;
+
+ if (!ItemIdIsDead(lp))
+ continue;
+
+ if (!ItemIdHasStorage(lp))
+ continue;
+
+ htup_begin = (char *) PageGetItem(page, lp);
+ memset(htup_begin, MASK_MARKER, ItemIdGetLength(lp));
+ }
+}
+
+/*
* Mask a heap page before performing consistency checks on it.
*/
void
@@ -9405,6 +9435,7 @@ heap_mask(char *pagedata, BlockNumber blkno)
mask_page_hint_bits(page);
mask_unused_space(page);
+ mask_dead_tuples(page);
for (off = 1; off <= PageGetMaxOffsetNumber(page); off++)
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343..ebbafe3 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -405,7 +408,7 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (HeapTupleSatisfiesVacuum(&tup, OldestXmin, buffer)
== HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
{
- heap_prune_record_unused(prstate, rootoffnum);
+ heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
ndeleted++;
@@ -580,8 +583,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
ndeleted++;
+ if (chainitems[i] == latestdead)
+ continue;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +603,28 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ /*
+ * We allow overlapping of redirected and dead items
+ */
+ heap_prune_record_redirect(prstate, rootoffnum, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
+ }
else
+ {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -653,7 +677,12 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
+
+ /*
+ * We suppress checking prstate->marked[offnum]. It is not the best idea,
+ * but this is most simplistic way to enable Dead Redirecting by
+ * overlapping Dead and Redirected states.
+ */
prstate->marked[offnum] = true;
}
@@ -706,7 +735,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 5649a70..15f7e32 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -139,7 +139,6 @@ typedef struct LVRelStats
bool lock_waiter_detected;
} LVRelStats;
-
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
@@ -156,6 +155,8 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
+static void quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -734,10 +735,17 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy)
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
+ }
/*
* Report that we are now vacuuming the heap. We also increase
* the number of index scans here; note that by using
@@ -1379,10 +1387,16 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy)
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
-
+ }
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
@@ -1671,6 +1685,152 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Get tuple from heap for a scan key building.
+ */
+static HeapTuple
+get_tuple_by_tid(Relation rel, ItemPointer tid)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ ItemId lp;
+ HeapTuple tuple;
+
+ buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
+ LockBuffer(buffer, BUFFER_LOCK_SHARE);
+
+ page = (Page) BufferGetPage(buffer);
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
+
+ /*
+ * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
+ */
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
+ /* Walk along the chain */
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+ Assert(ItemIdIsUsed(lp));
+ }
+
+ /* Form a tuple */
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(rel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ UnlockReleaseBuffer(buffer);
+ return tuple;
+}
+
+#include "access/nbtree.h"
+#include "catalog/index.h"
+#include "executor/executor.h"
+
+static int tid_comparator(const void* a, const void* b)
+{
+ return ItemPointerCompare((ItemPointer)a, (ItemPointer)b);
+}
+
+/*
+ * quick_vacuum_index() -- quick vacuum one index relation.
+ *
+ * Delete all the index entries pointing to tuples listed in
+ * vacrelstats->dead_tuples.
+ */
+static void
+quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats)
+{
+ IndexTargetDeleteResult stats;
+ IndexTargetDeleteInfo ivinfo;
+
+ if (irel->rd_amroutine->amtargetdelete != NULL)
+ {
+ int tnum;
+ bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
+ IndexInfo* indexInfo = BuildIndexInfo(irel);
+ EState* estate = CreateExecutorState();
+ ExprContext* econtext = GetPerTupleExprContext(estate);
+ ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+ qsort((void *)vacrelstats->dead_tuples, vacrelstats->num_dead_tuples, sizeof(ItemPointerData), tid_comparator);
+ ivinfo.isSorted = true;
+
+ /* Get tuple from heap */
+ for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ {
+ HeapTuple tuple;
+ TupleTableSlot* slot;
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+
+ /* Index entry for the TID was deleted early */
+ if (found[tnum])
+ continue;
+
+ /* Get a tuple from heap */
+ if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ {
+ /*
+ * Tuple has 'not used' status.
+ */
+ found[tnum] = true;
+ continue;
+ }
+
+ /*
+ * Form values[] and isnull[] arrays from for index tuple
+ * by heap tuple
+ */
+ slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ econtext->ecxt_scantuple = slot;
+
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * In a partial index, ignore tuples that don't satisfy the
+ * predicate.
+ */
+ if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
+ {
+ found[tnum] = true;
+ continue;
+ }
+
+ FormIndexDatum(indexInfo, slot, estate, values, isnull);
+
+ ExecDropSingleTupleTableSlot(slot);
+
+ /*
+ * Make attempt to delete some index entries by one tree descent.
+ * We use only a part of TID list, which contains not found TID's.
+ */
+ ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
+ ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples-tnum;
+ ivinfo.found_dead_tuples = found+tnum;
+ index_target_delete(&ivinfo, &stats, values, isnull);
+ }
+
+ pfree(found);
+ FreeExecutorState(estate);
+ }
+ else
+ {
+ IndexBulkDeleteResult **stats;
+
+ lazy_vacuum_index(irel, stats, vacrelstats);
+ }
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fa3c8a7..3f6c83b 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3236,7 +3236,15 @@ static struct config_real ConfigureNamesReal[] =
0.1, 0.0, 100.0,
NULL, NULL, NULL
},
-
+ {
+ {"target_index_deletion_factor", PGC_SIGHUP, AUTOVACUUM,
+ gettext_noop("Maximum number of vacuumed tuples as a fraction of reltuples where we can use target index vacuum strategy."),
+ NULL
+ },
+ &target_index_deletion_factor,
+ 0.01, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
{
{"checkpoint_completion_target", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval."),
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7..c5a7bc8 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,7 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
/*
* HeapScanIsValid
* True iff the heap scan is valid.
--
2.7.4
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-28 00:00 ` Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2 siblings, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-28 00:00 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> I still believe that the patch for physical TID ordering in btree:
> 1) has its own value, not only for target deletion,
> 2) will require only a few local changes in my code,
> and this patches can be developed independently.
I want to be clear on something now: I just don't think that this
patch has any chance of getting committed without something like my
own patch to go with it. The worst case for your patch without that
component is completely terrible. It's not really important for you to
actually formally make it part of your patch, so I'm not going to
insist on that or anything, but the reality is that my patch does not
have independent value -- and neither does yours.
I'm sorry if that sounds harsh, but this is a difficult, complicated
project. It's better to be clear about this stuff earlier on.
> I prepare third version of the patches. Summary:
> 1. Mask DEAD tuples at a page during consistency checking (See comments for
> the mask_dead_tuples() function).
> 2. Still not using physical TID ordering.
> 3. Index cleanup() after each quick_vacuum_index() call was excluded.
How does this patch affect opportunistic pruning in particular? Not
being able to immediately reclaim tuple space in the event of a dead
hot chain that is marked LP_DEAD could hurt quite a lot, including
with very common workloads, such as pgbench (pgbench accounts tuples
are quite a lot wider than a raw item pointer, and opportunistic
pruning is much more important than vacuuming). Is that going to be
acceptable, do you think? Have you measured the effects? Can we do
something about it, like make pruning behave differently when it's
opportunistic?
Are you aware of the difference between _bt_delitems_delete() and
_bt_delitems_vacuum(), and the considerations for hot standby? I think
that that's another TODO list item for this patch.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-28 05:36 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
0 siblings, 1 reply; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-28 05:36 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On 28.06.2018 05:00, Peter Geoghegan wrote:
> On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> I still believe that the patch for physical TID ordering in btree:
>> 1) has its own value, not only for target deletion,
>> 2) will require only a few local changes in my code,
>> and this patches can be developed independently.
>
> I want to be clear on something now: I just don't think that this
> patch has any chance of getting committed without something like my
> own patch to go with it. The worst case for your patch without that
> component is completely terrible. It's not really important for you to
> actually formally make it part of your patch, so I'm not going to
> insist on that or anything, but the reality is that my patch does not
> have independent value -- and neither does yours.
>
As I wrote before in the last email, I will integrate TID sorting to my
patches right now. Please, give me access to the last version of your
code, if it possible.
You can track the progress at https://github.com/danolivo/postgres git
repository
> I'm sorry if that sounds harsh, but this is a difficult, complicated
> project. It's better to be clear about this stuff earlier on.
Ok. It is clear now.
>
>> I prepare third version of the patches. Summary:
>> 1. Mask DEAD tuples at a page during consistency checking (See comments for
>> the mask_dead_tuples() function).
>> 2. Still not using physical TID ordering.
>> 3. Index cleanup() after each quick_vacuum_index() call was excluded.
>
> How does this patch affect opportunistic pruning in particular? Not
> being able to immediately reclaim tuple space in the event of a dead
> hot chain that is marked LP_DEAD could hurt quite a lot, including
> with very common workloads, such as pgbench (pgbench accounts tuples
> are quite a lot wider than a raw item pointer, and opportunistic
> pruning is much more important than vacuuming). Is that going to be
> acceptable, do you think? Have you measured the effects? Can we do
> something about it, like make pruning behave differently when it's
> opportunistic?
This is the most "tasty" part of the work. I plan some experimental
research on it at the end of patches developing (including TID sort)
and parametrized opportunistic pruning for flexibility of switching
between strategies on the fly.
My current opinion on this question: we can develop flexible strategy
based on parameters: free space at a block, frequency of UPDATE/DELETE
queries, percent of DEAD tuples in a block/relation.
Background cleaner, raised by heap_page_prune(), give an opportunity for
using different ways for each block or relation.
This technique should be able to configure from fully non-storage DEAD
tuples+vacuum to all-storage DEAD tuples+target deletion by DB admin.
>
> Are you aware of the difference between _bt_delitems_delete() and
> _bt_delitems_vacuum(), and the considerations for hot standby? I think
> that that's another TODO list item for this patch.
>
Ok
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-29 09:07 ` Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 1 reply; 47+ messages in thread
From: Юрий Соколов @ 2018-06-29 09:07 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
чт, 28 июн. 2018 г., 8:37 Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>:
>
>
> On 28.06.2018 05:00, Peter Geoghegan wrote:
> > On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov
> > <a.lepikhov@postgrespro.ru> wrote:
> >> I still believe that the patch for physical TID ordering in btree:
> >> 1) has its own value, not only for target deletion,
> >> 2) will require only a few local changes in my code,
> >> and this patches can be developed independently.
> >
> > I want to be clear on something now: I just don't think that this
> > patch has any chance of getting committed without something like my
> > own patch to go with it. The worst case for your patch without that
> > component is completely terrible. It's not really important for you to
> > actually formally make it part of your patch, so I'm not going to
> > insist on that or anything, but the reality is that my patch does not
> > have independent value -- and neither does yours.
> >
> As I wrote before in the last email, I will integrate TID sorting to my
> patches right now. Please, give me access to the last version of your
> code, if it possible.
> You can track the progress at https://github.com/danolivo/postgres git
> repository
Peter is absolutely right, imho: tie-breaking by TID within index
ordering is inevitable for reliable performance of this patch.
With regards,
Sokolov Yura.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
@ 2018-07-02 14:29 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-07-02 14:29 UTC (permalink / raw)
To: Юрий Соколов <funny.falcon@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On 29.06.2018 14:07, Юрий Соколов wrote:
> чт, 28 июн. 2018 г., 8:37 Andrey V. Lepikhov <a.lepikhov@postgrespro.ru
> <mailto:a.lepikhov@postgrespro.ru>>:
>
>
>
> On 28.06.2018 05:00, Peter Geoghegan wrote:
> > On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov
> > <a.lepikhov@postgrespro.ru <mailto:a.lepikhov@postgrespro.ru>> wrote:
> >> I still believe that the patch for physical TID ordering in btree:
> >> 1) has its own value, not only for target deletion,
> >> 2) will require only a few local changes in my code,
> >> and this patches can be developed independently.
> >
> > I want to be clear on something now: I just don't think that this
> > patch has any chance of getting committed without something like my
> > own patch to go with it. The worst case for your patch without that
> > component is completely terrible. It's not really important for
> you to
> > actually formally make it part of your patch, so I'm not going to
> > insist on that or anything, but the reality is that my patch does not
> > have independent value -- and neither does yours.
> >
> As I wrote before in the last email, I will integrate TID sorting to my
> patches right now. Please, give me access to the last version of your
> code, if it possible.
> You can track the progress at https://github.com/danolivo/postgres git
> repository
>
>
> Peter is absolutely right, imho: tie-breaking by TID within index
> ordering is inevitable for reliable performance of this patch.
>
In the new version the patch [1] was used in cooperation with 'retail
indextuple deletion' and 'quick vacuum strategy' patches (see
'0004-Retail-IndexTuple-Deletion-with-TID-sorting-in-leaf-.patch'.
To demonstrate the potential benefits, I did a test:
CREATE TABLE test (id serial primary key, value integer, factor integer);
INSERT INTO test (value, factor) SELECT random()*1e5, random()*1e3 FROM
generate_series(1, 1e7);
CREATE INDEX ON test(value);
VACUUM;
DELETE FROM test WHERE (factor = 1);
VACUUM test;
Execution time of last "VACUUM test;" command on my notebook was:
with bulk deletion: 1.6 s;
with Quick Vacuum Strategy: 5.2 s;
with Quick Vacuum Strategy & TID sorting: 0.6 s.
[1]
https://www.postgresql.org/message-id/CAH2-WzkVb0Kom%3DR%2B88fDFb%3DJSxZMFvbHVC6Mn9LJ2n%3DX%3DkS-Uw%...
> With regards,
> Sokolov Yura.
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
Attachments: download all (4 files, .tar.gz)
0001-Retail-IndexTuple-Deletion-Access-Method.patch (text/x-patch, 11.9K) download
From 7f2384691de592bf4d11cc8b4d75eca5500cd500 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 2 Jul 2018 16:13:08 +0500
Subject: [PATCH 1/4] Retail-IndexTuple-Deletion-Access-Method
---
contrib/bloom/blutils.c | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 15 ++++
src/backend/access/nbtree/nbtree.c | 138 +++++++++++++++++++++++++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/include/access/amapi.h | 6 ++
src/include/access/genam.h | 18 +++++
src/include/access/nbtree.h | 4 +
11 files changed, 187 insertions(+)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3..96f1d47 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbc..a0e06bd 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182..acf14e7 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42eff..d7a53d2 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df3..5fb32d6 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 22b5cc9..9ebeb78 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -730,6 +730,21 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexTargetDeleteResult *
+index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation indexRelation = info->indexRelation;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, values, isnull);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 27a3032..c54aeac 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -34,7 +34,9 @@
#include "storage/smgr.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/syscache.h"
/* Working state needed by btvacuumpage */
@@ -127,6 +129,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -884,6 +887,141 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+static int
+tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
+{
+ for (int i = 0; i < ntid; i++)
+ if (ItemPointerEquals(tid, &(tid_list[i])))
+ return i;
+ return -1;
+}
+
+IndexTargetDeleteResult*
+bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation irel = info->indexRelation;
+ Relation hrel = info->heapRelation;
+ ScanKey skey;
+ int keysCount = IndexRelationGetNumberOfKeyAttributes(irel);
+ BTStack stack;
+ Buffer buf;
+ Page page;
+ BTPageOpaque opaque;
+ OffsetNumber offnum;
+ int ndeletable = 0;
+ OffsetNumber deletable[MaxOffsetNumber];
+ IndexTuple itup;
+
+ if (stats == NULL)
+ stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+
+ itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
+ skey = _bt_mkscankey(irel, itup);
+
+ /* Descend the tree and position ourselves on the target leaf page. */
+ stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
+ _bt_freestack(stack);
+
+ /* To prepare tuple entries search across index pages */
+ Assert(BufferIsValid(buf));
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ for (;;)
+ {
+ int32 cmpval;
+ ItemId itemid;
+ IndexTuple itup;
+ int pos;
+
+ /* Switch to the next page */
+ if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ {
+ /*
+ * Before unlocking index page we need to delete
+ * all currently found tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ ndeletable = 0;
+ }
+
+ /*
+ * Check for end-of-index
+ */
+ if (P_RIGHTMOST(opaque))
+ /* it is rightmost leaf */
+ break;
+
+ /*
+ * Switch to the next index page
+ */
+ buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = P_FIRSTDATAKEY(opaque);
+ continue;
+ }
+
+ /*
+ * This index entry satisfied to the scan key?
+ */
+ cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+
+ if (cmpval != 0)
+ /* End of index entries, satisfied to the scan key */
+ break;
+
+ /*
+ * To load index tuple and look for matches in the TID list of
+ * dead heap tuples
+ */
+ itemid = PageGetItemId(page, offnum);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
+
+ if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+
+ offnum = OffsetNumberNext(offnum);
+ }
+
+ /*
+ * Delete all found index tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ }
+
+ /* Release scan key, unpin and unlock buffer */
+ _bt_freeskey(skey);
+ _bt_relbuf(irel, buf);
+
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4a9b5da..3257281 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -56,6 +56,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6..497b54e 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,11 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexTargetDeleteResult *(*amtargetdelete_function) (IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 24c720b..79b386e 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -51,6 +51,15 @@ typedef struct IndexVacuumInfo
BufferAccessStrategy strategy; /* access strategy for reads */
} IndexVacuumInfo;
+typedef struct IndexTargetDeleteInfo
+{
+ Relation heapRelation;
+ Relation indexRelation; /* the index being vacuumed */
+ int num_dead_tuples;
+ ItemPointer dead_tuples;
+ bool* found_dead_tuples;
+} IndexTargetDeleteInfo;
+
/*
* Struct for statistics returned by ambulkdelete and amvacuumcleanup
*
@@ -79,6 +88,11 @@ typedef struct IndexBulkDeleteResult
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+typedef struct IndexTargetDeleteResult
+{
+ int tuples_removed; /* # removed during vacuum operation */
+} IndexTargetDeleteResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
@@ -163,6 +177,10 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexTargetDeleteResult *index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 04ecb4c..3b9461c 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -508,6 +508,10 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexTargetDeleteResult *bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
--
2.7.4
0002-Quick-vacuum-strategy.patch (text/x-patch, 12.3K) download
From 904d204c36a46d9f8d7433af37bbd76031516ad7 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 2 Jul 2018 16:17:01 +0500
Subject: [PATCH 2/4] Quick-vacuum-strategy
---
src/backend/access/heap/heapam.c | 31 +++++++
src/backend/access/heap/pruneheap.c | 40 ++++++++-
src/backend/commands/vacuumlazy.c | 161 +++++++++++++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 10 ++-
src/include/access/heapam.h | 1 +
5 files changed, 234 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 72395a5..dd03366 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -9393,6 +9393,36 @@ heap_sync(Relation rel)
}
/*
+ * Mask DEAD tuples at a HEAP page
+ *
+ * We introduce this function for wal_consistency_checking satisfaction at
+ * Hot Standby node.
+ *
+ * DEAD tuple on a master node has t_cid and t_infomask, which can not be
+ * obtained by WAL records applying on a Hot Standby node.
+ */
+static void
+mask_dead_tuples(Page page)
+{
+ OffsetNumber offnum;
+
+ for (offnum = FirstOffsetNumber; offnum <= PageGetMaxOffsetNumber(page); offnum = OffsetNumberNext(offnum))
+ {
+ ItemId lp = PageGetItemId(page, offnum);
+ char* htup_begin;
+
+ if (!ItemIdIsDead(lp))
+ continue;
+
+ if (!ItemIdHasStorage(lp))
+ continue;
+
+ htup_begin = (char *) PageGetItem(page, lp);
+ memset(htup_begin, MASK_MARKER, ItemIdGetLength(lp));
+ }
+}
+
+/*
* Mask a heap page before performing consistency checks on it.
*/
void
@@ -9405,6 +9435,7 @@ heap_mask(char *pagedata, BlockNumber blkno)
mask_page_hint_bits(page);
mask_unused_space(page);
+ mask_dead_tuples(page);
for (off = 1; off <= PageGetMaxOffsetNumber(page); off++)
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343..ebbafe3 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -405,7 +408,7 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (HeapTupleSatisfiesVacuum(&tup, OldestXmin, buffer)
== HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
{
- heap_prune_record_unused(prstate, rootoffnum);
+ heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
ndeleted++;
@@ -580,8 +583,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
ndeleted++;
+ if (chainitems[i] == latestdead)
+ continue;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +603,28 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ /*
+ * We allow overlapping of redirected and dead items
+ */
+ heap_prune_record_redirect(prstate, rootoffnum, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
+ }
else
+ {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -653,7 +677,12 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
+
+ /*
+ * We suppress checking prstate->marked[offnum]. It is not the best idea,
+ * but this is most simplistic way to enable Dead Redirecting by
+ * overlapping Dead and Redirected states.
+ */
prstate->marked[offnum] = true;
}
@@ -706,7 +735,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 5649a70..0bb8abb 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -41,13 +41,16 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/nbtree.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
+#include "catalog/index.h"
#include "catalog/storage.h"
#include "commands/dbcommands.h"
#include "commands/progress.h"
#include "commands/vacuum.h"
+#include "executor/executor.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "portability/instr_time.h"
@@ -139,7 +142,6 @@ typedef struct LVRelStats
bool lock_waiter_detected;
} LVRelStats;
-
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
@@ -156,6 +158,8 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
+static void quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -734,10 +738,17 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples / vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
+ }
/*
* Report that we are now vacuuming the heap. We also increase
* the number of index scans here; note that by using
@@ -1379,10 +1390,16 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples / vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
-
+ }
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
@@ -1671,6 +1688,142 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Get tuple from heap for a scan key building.
+ */
+static HeapTuple
+get_tuple_by_tid(Relation rel, ItemPointer tid)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ ItemId lp;
+ HeapTuple tuple;
+ bool needLock = !RELATION_IS_LOCAL(rel);
+ BlockNumber npages;
+
+ if (needLock)
+ LockRelationForExtension(rel, ExclusiveLock);
+ npages = RelationGetNumberOfBlocks(rel);
+ if (needLock)
+ UnlockRelationForExtension(rel, ExclusiveLock);
+ if (ItemPointerGetBlockNumber(tid) > npages)
+ return NULL;
+
+ buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
+ LockBuffer(buffer, BUFFER_LOCK_SHARE);
+
+ page = (Page) BufferGetPage(buffer);
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
+
+ /*
+ * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
+ */
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
+ /* Walk along the chain */
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+ Assert(ItemIdIsUsed(lp));
+ }
+
+ /* Form a tuple */
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(rel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ UnlockReleaseBuffer(buffer);
+ return tuple;
+}
+
+/*
+ * quick_vacuum_index() -- quick vacuum one index relation.
+ *
+ * Delete all the index entries pointing to tuples listed in
+ * vacrelstats->dead_tuples.
+ */
+static void
+quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats)
+{
+ int tnum;
+ bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
+ IndexInfo* indexInfo = BuildIndexInfo(irel);
+ EState* estate = CreateExecutorState();
+ ExprContext* econtext = GetPerTupleExprContext(estate);
+ ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+
+ IndexTargetDeleteResult stats;
+ IndexTargetDeleteInfo ivinfo;
+
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+
+ econtext->ecxt_scantuple = slot;
+
+ /* Get tuple from heap */
+ for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ {
+ HeapTuple tuple;
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+
+ /* Index entry for the TID was deleted early */
+ if (found[tnum])
+ continue;
+
+ /* Get a tuple from heap */
+ if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ {
+ /*
+ * Tuple has 'not used' status.
+ */
+ found[tnum] = true;
+ continue;
+ }
+
+ /*
+ * Form values[] and isnull[] arrays from for index tuple
+ * by heap tuple
+ */
+ MemoryContextReset(econtext->ecxt_per_tuple_memory);
+
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * In a partial index, ignore tuples that don't satisfy the
+ * predicate.
+ */
+ if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
+ {
+ found[tnum] = true;
+ continue;
+ }
+
+ FormIndexDatum(indexInfo, slot, estate, values, isnull);
+
+ /*
+ * Make attempt to delete some index entries by one tree descent.
+ * We use only a part of TID list, which contains not found TID's.
+ */
+ ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
+ ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples - tnum;
+ ivinfo.found_dead_tuples = found + tnum;
+ index_target_delete(&ivinfo, &stats, values, isnull);
+ }
+
+ pfree(found);
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fa3c8a7..3f6c83b 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3236,7 +3236,15 @@ static struct config_real ConfigureNamesReal[] =
0.1, 0.0, 100.0,
NULL, NULL, NULL
},
-
+ {
+ {"target_index_deletion_factor", PGC_SIGHUP, AUTOVACUUM,
+ gettext_noop("Maximum number of vacuumed tuples as a fraction of reltuples where we can use target index vacuum strategy."),
+ NULL
+ },
+ &target_index_deletion_factor,
+ 0.01, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
{
{"checkpoint_completion_target", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval."),
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7..c5a7bc8 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,7 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
/*
* HeapScanIsValid
* True iff the heap scan is valid.
--
2.7.4
0003-Ensure-nbtree-leaf-tuple-keys-are-always-unique.patch (text/x-patch, 96.9K) download
From 612a97598317e81d1087413d1cd10f3c5b6cc44a Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 2 Jul 2018 16:26:56 +0500
Subject: [PATCH 3/4] Ensure-nbtree-leaf-tuple-keys-are-always-unique
---
contrib/amcheck/verify_nbtree.c | 219 +++++++++++++++------
contrib/earthdistance/expected/earthdistance.out | 2 +-
contrib/file_fdw/output/file_fdw.source | 10 +-
src/backend/access/nbtree/README | 70 ++++---
src/backend/access/nbtree/nbtinsert.c | 133 ++++++++-----
src/backend/access/nbtree/nbtpage.c | 6 +-
src/backend/access/nbtree/nbtsearch.c | 96 +++++++--
src/backend/access/nbtree/nbtsort.c | 53 +++--
src/backend/access/nbtree/nbtutils.c | 155 ++++++++++++---
src/backend/access/nbtree/nbtxlog.c | 3 +
src/backend/utils/sort/tuplesort.c | 7 +-
src/include/access/nbtree.h | 71 +++++--
.../test_extensions/expected/test_extensions.out | 6 +-
src/test/regress/expected/aggregates.out | 4 +-
src/test/regress/expected/alter_table.out | 20 +-
src/test/regress/expected/collate.out | 2 +-
src/test/regress/expected/create_type.out | 8 +-
src/test/regress/expected/create_view.out | 2 +-
src/test/regress/expected/dependency.out | 4 +-
src/test/regress/expected/domain.out | 4 +-
src/test/regress/expected/event_trigger.out | 75 ++++---
src/test/regress/expected/foreign_data.out | 8 +-
src/test/regress/expected/foreign_key.out | 2 +-
src/test/regress/expected/indexing.out | 12 +-
src/test/regress/expected/inherit.out | 16 +-
src/test/regress/expected/matview.out | 18 +-
src/test/regress/expected/rowsecurity.out | 4 +-
src/test/regress/expected/select_into.out | 4 +-
src/test/regress/expected/triggers.out | 16 +-
src/test/regress/expected/truncate.out | 4 +-
src/test/regress/expected/typed_table.out | 12 +-
src/test/regress/expected/updatable_views.out | 28 +--
src/test/regress/output/tablespace.source | 8 +-
src/test/regress/sql/updatable_views.sql | 2 +
34 files changed, 731 insertions(+), 353 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index a1438a2..2358bfa 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -46,6 +46,13 @@ PG_MODULE_MAGIC;
#define InvalidBtreeLevel ((uint32) InvalidBlockNumber)
/*
+ * Convenience macro to get number of key attributes in tuple in low-context
+ * fashion
+ */
+#define BTreeTupleGetNKeyAtts(itup, rel) \
+ Min(IndexRelationGetNumberOfKeyAttributes(rel), BTreeTupleGetNAtts(itup, rel))
+
+/*
* State associated with verifying a B-Tree index
*
* target is the point of reference for a verification operation.
@@ -125,25 +132,27 @@ static void bt_check_every_level(Relation rel, Relation heaprel,
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_target_page_check(BtreeCheckState *state);
-static ScanKey bt_right_page_check_scankey(BtreeCheckState *state);
+static IndexTuple bt_right_page_check_tuple(BtreeCheckState *state);
static void bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
- ScanKey targetkey);
+ ScanKey targetkey, ItemPointer scantid, int tupnkeyatts);
static void bt_downlink_missing_check(BtreeCheckState *state);
static void bt_tuple_present_callback(Relation index, HeapTuple htup,
Datum *values, bool *isnull,
bool tupleIsAlive, void *checkstate);
static inline bool offset_is_negative_infinity(BTPageOpaque opaque,
OffsetNumber offset);
+static inline bool invariant_l_offset(BtreeCheckState *state,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
+ OffsetNumber upperbound);
static inline bool invariant_leq_offset(BtreeCheckState *state,
- ScanKey key,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
OffsetNumber upperbound);
-static inline bool invariant_geq_offset(BtreeCheckState *state,
- ScanKey key,
+static inline bool invariant_g_offset(BtreeCheckState *state,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
OffsetNumber lowerbound);
-static inline bool invariant_leq_nontarget_offset(BtreeCheckState *state,
- Page other,
- ScanKey key,
- OffsetNumber upperbound);
+static inline bool invariant_l_nontarget_offset(BtreeCheckState *state,
+ Page other, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber upperbound);
static Page palloc_btree_page(BtreeCheckState *state, BlockNumber blocknum);
/*
@@ -834,8 +843,10 @@ bt_target_page_check(BtreeCheckState *state)
{
ItemId itemid;
IndexTuple itup;
- ScanKey skey;
size_t tupsize;
+ int tupnkeyatts;
+ ScanKey skey;
+ ItemPointer scantid;
CHECK_FOR_INTERRUPTS();
@@ -902,8 +913,17 @@ bt_target_page_check(BtreeCheckState *state)
if (offset_is_negative_infinity(topaque, offset))
continue;
- /* Build insertion scankey for current page offset */
+ /*
+ * Build insertion scankey for current page offset/tuple.
+ *
+ * As required by _bt_mkscankey(), track number of key attributes,
+ * which is needed so that _bt_compare() calls handle truncated
+ * attributes correctly. Never count non-key attributes in
+ * non-truncated tuples as key attributes, though.
+ */
+ tupnkeyatts = BTreeTupleGetNKeyAtts(itup, state->rel);
skey = _bt_mkscankey(state->rel, itup);
+ scantid = BTreeTupleGetHeapTID(itup);
/* Fingerprint leaf page tuples (those that point to the heap) */
if (state->heapallindexed && P_ISLEAF(topaque) && !ItemIdIsDead(itemid))
@@ -930,7 +950,7 @@ bt_target_page_check(BtreeCheckState *state)
* and probably not markedly more effective in practice.
*/
if (!P_RIGHTMOST(topaque) &&
- !invariant_leq_offset(state, skey, P_HIKEY))
+ !invariant_leq_offset(state, tupnkeyatts, skey, scantid, P_HIKEY))
{
char *itid,
*htid;
@@ -956,11 +976,11 @@ bt_target_page_check(BtreeCheckState *state)
* * Item order check *
*
* Check that items are stored on page in logical order, by checking
- * current item is less than or equal to next item (if any).
+ * current item is strictly less than next item (if any).
*/
if (OffsetNumberNext(offset) <= max &&
- !invariant_leq_offset(state, skey,
- OffsetNumberNext(offset)))
+ !invariant_l_offset(state, tupnkeyatts, skey, scantid,
+ OffsetNumberNext(offset)))
{
char *itid,
*htid,
@@ -1017,16 +1037,28 @@ bt_target_page_check(BtreeCheckState *state)
*/
else if (offset == max)
{
+ IndexTuple righttup;
ScanKey rightkey;
+ int righttupnkeyatts;
+ ItemPointer rightscantid;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ righttup = bt_right_page_check_tuple(state);
+
+ /* Set up right item scankey */
+ if (righttup)
+ {
+ righttupnkeyatts = BTreeTupleGetNKeyAtts(righttup, state->rel);
+ rightkey = _bt_mkscankey(state->rel, righttup);
+ rightscantid = BTreeTupleGetHeapTID(righttup);
+ }
- if (rightkey &&
- !invariant_geq_offset(state, rightkey, max))
+ if (righttup &&
+ !invariant_g_offset(state, righttupnkeyatts, rightkey,
+ rightscantid, max))
{
/*
- * As explained at length in bt_right_page_check_scankey(),
+ * As explained at length in bt_right_page_check_tuple(),
* there is a known !readonly race that could account for
* apparent violation of invariant, which we must check for
* before actually proceeding with raising error. Our canary
@@ -1069,7 +1101,7 @@ bt_target_page_check(BtreeCheckState *state)
{
BlockNumber childblock = BTreeInnerTupleGetDownLink(itup);
- bt_downlink_check(state, childblock, skey);
+ bt_downlink_check(state, childblock, skey, scantid, tupnkeyatts);
}
}
@@ -1083,9 +1115,9 @@ bt_target_page_check(BtreeCheckState *state)
}
/*
- * Return a scankey for an item on page to right of current target (or the
+ * Return an index tuple for an item on page to right of current target (or the
* first non-ignorable page), sufficient to check ordering invariant on last
- * item in current target page. Returned scankey relies on local memory
+ * item in current target page. Returned tuple relies on local memory
* allocated for the child page, which caller cannot pfree(). Caller's memory
* context should be reset between calls here.
*
@@ -1098,8 +1130,8 @@ bt_target_page_check(BtreeCheckState *state)
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
*/
-static ScanKey
-bt_right_page_check_scankey(BtreeCheckState *state)
+static IndexTuple
+bt_right_page_check_tuple(BtreeCheckState *state)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1287,11 +1319,10 @@ bt_right_page_check_scankey(BtreeCheckState *state)
}
/*
- * Return first real item scankey. Note that this relies on right page
- * memory remaining allocated.
+ * Return first real item. Note that this relies on right page memory
+ * remaining allocated.
*/
- return _bt_mkscankey(state->rel,
- (IndexTuple) PageGetItem(rightpage, rightitem));
+ return (IndexTuple) PageGetItem(rightpage, rightitem);
}
/*
@@ -1305,7 +1336,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
*/
static void
bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
- ScanKey targetkey)
+ ScanKey targetkey, ItemPointer scantid, int tupnkeyatts)
{
OffsetNumber offset;
OffsetNumber maxoffset;
@@ -1354,7 +1385,7 @@ bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
/*
* Verify child page has the downlink key from target page (its parent) as
- * a lower bound.
+ * a lower bound; downlink must be strictly less than all keys on the page.
*
* Check all items, rather than checking just the first and trusting that
* the operator class obeys the transitive law.
@@ -1404,14 +1435,14 @@ bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
/*
* Skip comparison of target page key against "negative infinity"
* item, if any. Checking it would indicate that it's not an upper
- * bound, but that's only because of the hard-coding within
- * _bt_compare().
+ * bound, but that's only because of the hard-coding for negative
+ * inifinity items within _bt_compare().
*/
if (offset_is_negative_infinity(copaque, offset))
continue;
- if (!invariant_leq_nontarget_offset(state, child,
- targetkey, offset))
+ if (!invariant_l_nontarget_offset(state, child, tupnkeyatts, targetkey,
+ scantid, offset))
ereport(ERROR,
(errcode(ERRCODE_INDEX_CORRUPTED),
errmsg("down-link lower bound invariant violated for index \"%s\"",
@@ -1752,6 +1783,51 @@ offset_is_negative_infinity(BTPageOpaque opaque, OffsetNumber offset)
}
/*
+ * Does the invariant hold that the key is strictly less than a given upper
+ * bound offset item?
+ *
+ * If this function returns false, convention is that caller throws error due
+ * to corruption.
+ */
+static inline bool
+invariant_l_offset(BtreeCheckState *state, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber upperbound)
+{
+ int32 cmp;
+
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, state->target,
+ upperbound);
+
+ /*
+ * _bt_compare interprets the absence of attributes in scan keys as meaning
+ * that they're not participating in a search, not as negative infinity
+ * (only tuples within the index are treated as negative infinity).
+ * Compensate for that here.
+ */
+ if (cmp == 0)
+ {
+ ItemId itemid;
+ IndexTuple ritup;
+ int uppnkeyatts;
+ ItemPointer rheaptid;
+
+ itemid = PageGetItemId(state->target, upperbound);
+ ritup = (IndexTuple) PageGetItem(state->target, itemid);
+ uppnkeyatts = BTreeTupleGetNKeyAtts(ritup, state->rel);
+
+ /* Get heap TID for item to the right */
+ rheaptid = BTreeTupleGetHeapTID(ritup);
+
+ if (uppnkeyatts == tupnkeyatts)
+ return scantid == NULL && rheaptid != NULL;
+
+ return tupnkeyatts < uppnkeyatts;
+ }
+
+ return cmp < 0;
+}
+
+/*
* Does the invariant hold that the key is less than or equal to a given upper
* bound offset item?
*
@@ -1759,57 +1835,90 @@ offset_is_negative_infinity(BTPageOpaque opaque, OffsetNumber offset)
* to corruption.
*/
static inline bool
-invariant_leq_offset(BtreeCheckState *state, ScanKey key,
- OffsetNumber upperbound)
+invariant_leq_offset(BtreeCheckState *state, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber upperbound)
{
- int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(state->rel);
int32 cmp;
- cmp = _bt_compare(state->rel, nkeyatts, key, state->target, upperbound);
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, state->target,
+ upperbound);
return cmp <= 0;
}
/*
- * Does the invariant hold that the key is greater than or equal to a given
- * lower bound offset item?
+ * Does the invariant hold that the key is strictly greater than a given lower
+ * bound offset item?
*
* If this function returns false, convention is that caller throws error due
* to corruption.
*/
static inline bool
-invariant_geq_offset(BtreeCheckState *state, ScanKey key,
- OffsetNumber lowerbound)
+invariant_g_offset(BtreeCheckState *state, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber lowerbound)
{
- int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(state->rel);
int32 cmp;
- cmp = _bt_compare(state->rel, nkeyatts, key, state->target, lowerbound);
+ /*
+ * No need to consider possibility that scankey has attributes that we need
+ * to force to be interpreted as negative infinity, since scan key has to
+ * be strictly greater than lower bound offset.
+ */
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, state->target,
+ lowerbound);
- return cmp >= 0;
+ return cmp > 0;
}
/*
- * Does the invariant hold that the key is less than or equal to a given upper
+ * Does the invariant hold that the key is strictly less than a given upper
* bound offset item, with the offset relating to a caller-supplied page that
- * is not the current target page? Caller's non-target page is typically a
- * child page of the target, checked as part of checking a property of the
- * target page (i.e. the key comes from the target).
+ * is not the current target page?
+ *
+ * Caller's non-target page is a child page of the target, checked as part of
+ * checking a property of the target page (i.e. the key comes from the
+ * target).
*
* If this function returns false, convention is that caller throws error due
* to corruption.
*/
static inline bool
-invariant_leq_nontarget_offset(BtreeCheckState *state,
- Page nontarget, ScanKey key,
- OffsetNumber upperbound)
+invariant_l_nontarget_offset(BtreeCheckState *state, Page nontarget,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
+ OffsetNumber upperbound)
{
- int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(state->rel);
int32 cmp;
- cmp = _bt_compare(state->rel, nkeyatts, key, nontarget, upperbound);
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, nontarget,
+ upperbound);
- return cmp <= 0;
+ /*
+ * _bt_compare interprets the absence of attributes in scan keys as meaning
+ * that they're not participating in a search, not as negative infinity
+ * (only tuples within the index are treated as negative infinity).
+ * Compensate for that here.
+ */
+ if (cmp == 0)
+ {
+ ItemId itemid;
+ IndexTuple child;
+ int uppnkeyatts;
+ ItemPointer childheaptid;
+
+ itemid = PageGetItemId(nontarget, upperbound);
+ child = (IndexTuple) PageGetItem(nontarget, itemid);
+ uppnkeyatts = BTreeTupleGetNKeyAtts(child, state->rel);
+
+ /* Get heap TID for item from child/non-target */
+ childheaptid = BTreeTupleGetHeapTID(child);
+
+ if (uppnkeyatts == tupnkeyatts)
+ return scantid == NULL && childheaptid != NULL;
+
+ return tupnkeyatts < uppnkeyatts;
+ }
+
+ return cmp < 0;
}
/*
diff --git a/contrib/earthdistance/expected/earthdistance.out b/contrib/earthdistance/expected/earthdistance.out
index 36a5a7b..0ce34da 100644
--- a/contrib/earthdistance/expected/earthdistance.out
+++ b/contrib/earthdistance/expected/earthdistance.out
@@ -968,7 +968,7 @@ SELECT abs(cube_distance(ll_to_earth(-30,-90), '(0)'::cube) / earth() - 1) <
drop extension cube; -- fail, earthdistance requires it
ERROR: cannot drop extension cube because other objects depend on it
-DETAIL: extension earthdistance depends on extension cube
+DETAIL: extension earthdistance depends on function cube_out(cube)
HINT: Use DROP ... CASCADE to drop the dependent objects too.
drop extension earthdistance;
drop type cube; -- fail, extension cube requires it
diff --git a/contrib/file_fdw/output/file_fdw.source b/contrib/file_fdw/output/file_fdw.source
index 853c9f9..42bf16b 100644
--- a/contrib/file_fdw/output/file_fdw.source
+++ b/contrib/file_fdw/output/file_fdw.source
@@ -432,10 +432,10 @@ RESET ROLE;
DROP EXTENSION file_fdw CASCADE;
NOTICE: drop cascades to 7 other objects
DETAIL: drop cascades to server file_server
-drop cascades to user mapping for regress_file_fdw_superuser on server file_server
-drop cascades to user mapping for regress_no_priv_user on server file_server
-drop cascades to foreign table agg_text
-drop cascades to foreign table agg_csv
-drop cascades to foreign table agg_bad
drop cascades to foreign table text_csv
+drop cascades to foreign table agg_bad
+drop cascades to foreign table agg_csv
+drop cascades to foreign table agg_text
+drop cascades to user mapping for regress_no_priv_user on server file_server
+drop cascades to user mapping for regress_file_fdw_superuser on server file_server
DROP ROLE regress_file_fdw_superuser, regress_file_fdw_user, regress_no_priv_user;
diff --git a/src/backend/access/nbtree/README b/src/backend/access/nbtree/README
index 3680e69..0782f01 100644
--- a/src/backend/access/nbtree/README
+++ b/src/backend/access/nbtree/README
@@ -46,18 +46,15 @@ the real "key" at all, just at the link field.) We can distinguish
items at the leaf level in the same way, by examining their links to
heap tuples; we'd never have two items for the same heap tuple.
-Lehman and Yao assume that the key range for a subtree S is described
+Lehman and Yao require that the key range for a subtree S is described
by Ki < v <= Ki+1 where Ki and Ki+1 are the adjacent keys in the parent
-page. This does not work for nonunique keys (for example, if we have
-enough equal keys to spread across several leaf pages, there *must* be
-some equal bounding keys in the first level up). Therefore we assume
-Ki <= v <= Ki+1 instead. A search that finds exact equality to a
-bounding key in an upper tree level must descend to the left of that
-key to ensure it finds any equal keys in the preceding page. An
-insertion that sees the high key of its target page is equal to the key
-to be inserted has a choice whether or not to move right, since the new
-key could go on either page. (Currently, we try to find a page where
-there is room for the new key without a split.)
+page. A search that finds exact equality to a bounding key in an upper
+tree level must descend to the left of that key to ensure it finds any
+equal keys. An insertion that sees the high key of its target page is
+equal to the key to be inserted cannot move right, since the downlink
+for the right sibling in the parent must always be strictly less than
+right sibling keys (this is always possible because the leftmost
+downlink on any non-leaf level is always a negative infinity downlink).
Lehman and Yao don't require read locks, but assume that in-memory
copies of tree pages are unshared. Postgres shares in-memory buffers
@@ -610,21 +607,25 @@ scanned to decide whether to return the entry and whether the scan can
stop (see _bt_checkkeys()).
We use term "pivot" index tuples to distinguish tuples which don't point
-to heap tuples, but rather used for tree navigation. Pivot tuples includes
-all tuples on non-leaf pages and high keys on leaf pages. Note that pivot
-index tuples are only used to represent which part of the key space belongs
-on each page, and can have attribute values copied from non-pivot tuples
-that were deleted and killed by VACUUM some time ago. In principle, we could
-truncate away attributes that are not needed for a page high key during a leaf
-page split, provided that the remaining attributes distinguish the last index
-tuple on the post-split left page as belonging on the left page, and the first
-index tuple on the post-split right page as belonging on the right page. This
-optimization is sometimes called suffix truncation, and may appear in a future
-release. Since the high key is subsequently reused as the downlink in the
-parent page for the new right page, suffix truncation can increase index
-fan-out considerably by keeping pivot tuples short. INCLUDE indexes similarly
-truncate away non-key attributes at the time of a leaf page split,
-increasing fan-out.
+to heap tuples, that are used only for tree navigation. Pivot tuples
+includes all tuples on non-leaf pages and high keys on leaf pages. Note
+that pivot index tuples are only used to represent which part of the key
+space belongs on each page, and can have attribute values copied from
+non-pivot tuples that were deleted and killed by VACUUM some time ago.
+
+We truncate away attributes that are not needed for a page high key during
+a leaf page split, provided that the remaining attributes distinguish the
+last index tuple on the post-split left page as belonging on the left
+page, and the first index tuple on the post-split right page as belonging
+on the right page. A truncated tuple logically retains the truncated
+suffix key attributes, which implicitly have "negative infinity" as their
+value. This optimization is called suffix truncation. Since the high key
+is subsequently reused as the downlink in the parent page for the new
+right page, suffix truncation can increase index fan-out considerably by
+keeping pivot tuples short. INCLUDE indexes are guaranteed to have
+non-key attributes truncated at the time of a leaf page split, but may
+also have some key attributes truncated away, based on the usual criteria
+for key attributes.
Notes About Data Representation
-------------------------------
@@ -658,4 +659,19 @@ downlink. The first data item on each such page has no lower bound
routines must treat it accordingly. The actual key stored in the
item is irrelevant, and need not be stored at all. This arrangement
corresponds to the fact that an L&Y non-leaf page has one more pointer
-than key.
+than key. Suffix truncation's negative infinity attributes behave in
+the same way.
+
+Non-leaf pages only truly need to truncate their first item to zero
+attributes at the leftmost level, since that truly is negative infinity.
+All other negative infinity items are only really negative infinity
+within the subtree that the page is at the root of (or is a leftmost
+page within). We truncate away all attributes of the first item on
+non-leaf pages just the same, to save a little space. If we ever
+avoided zero-truncating items on pages where that doesn't accurately
+represent the absolute separation of the keyspace, we'd be left with
+"low key" items on internal pages -- a key value that can be used as a
+lower bound on items on the page, much like the high key is an upper
+bound. (Actually, that would even be true of "true" negative infinity
+items. One can think of rightmost pages as implicitly containing
+"positive infinity" high keys.)
diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c
index 907cce0..4c4f7d8 100644
--- a/src/backend/access/nbtree/nbtinsert.c
+++ b/src/backend/access/nbtree/nbtinsert.c
@@ -180,7 +180,7 @@ top:
!P_IGNORE(lpageop) &&
(PageGetFreeSpace(page) > itemsz) &&
PageGetMaxOffsetNumber(page) >= P_FIRSTDATAKEY(lpageop) &&
- _bt_compare(rel, indnkeyatts, itup_scankey, page,
+ _bt_compare(rel, indnkeyatts, itup_scankey, &itup->t_tid, page,
P_FIRSTDATAKEY(lpageop)) > 0)
{
/*
@@ -216,9 +216,12 @@ top:
if (!fastpath)
{
+ ItemPointer scantid =
+ (checkUnique != UNIQUE_CHECK_NO ? NULL : &itup->t_tid);
+
/* find the first page containing this key */
- stack = _bt_search(rel, indnkeyatts, itup_scankey, false, &buf, BT_WRITE,
- NULL);
+ stack = _bt_search(rel, indnkeyatts, itup_scankey, scantid, false,
+ &buf, BT_WRITE, NULL);
/* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
@@ -231,8 +234,8 @@ top:
* need to move right in the tree. See Lehman and Yao for an
* excruciatingly precise description.
*/
- buf = _bt_moveright(rel, buf, indnkeyatts, itup_scankey, false,
- true, stack, BT_WRITE, NULL);
+ buf = _bt_moveright(rel, buf, indnkeyatts, itup_scankey, scantid,
+ false, true, stack, BT_WRITE, NULL);
}
/*
@@ -261,7 +264,8 @@ top:
TransactionId xwait;
uint32 speculativeToken;
- offset = _bt_binsrch(rel, buf, indnkeyatts, itup_scankey, false);
+ /* Find position while excluding heap TID attribute */
+ offset = _bt_binsrch(rel, buf, indnkeyatts, itup_scankey, NULL, false);
xwait = _bt_check_unique(rel, itup, heapRel, buf, offset, itup_scankey,
checkUnique, &is_unique, &speculativeToken);
@@ -285,6 +289,25 @@ top:
_bt_freestack(stack);
goto top;
}
+
+ /*
+ * Be careful to not get confused about user attribute position and
+ * insertion position.
+ *
+ * XXX: This is ugly as sin, and clearly needs a lot more work. While
+ * not having this code does not seem to affect regression tests, we
+ * almost certainly need to do something here for the case where
+ * _bt_check_unique() traverses many pages, each filled with logical
+ * duplicates.
+ */
+ buf = _bt_moveright(rel, buf, indnkeyatts, itup_scankey, &itup->t_tid,
+ false, true, stack, BT_WRITE, NULL);
+ /*
+ * Always invalidate hint
+ *
+ * FIXME: This is unacceptable.
+ */
+ offset = InvalidOffsetNumber;
}
if (checkUnique != UNIQUE_CHECK_EXISTING)
@@ -564,11 +587,11 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
offset = OffsetNumberNext(offset);
else
{
- /* If scankey == hikey we gotta check the next page too */
+ /* If scankey <= hikey we gotta check the next page too */
if (P_RIGHTMOST(opaque))
break;
- if (!_bt_isequal(itupdesc, page, P_HIKEY,
- indnkeyatts, itup_scankey))
+ /* _bt_isequal()'s special NULL semantics not required here */
+ if (_bt_compare(rel, indnkeyatts, itup_scankey, NULL, page, P_HIKEY) > 0)
break;
/* Advance to next non-dead page --- there must be one */
for (;;)
@@ -700,6 +723,18 @@ _bt_findinsertloc(Relation rel,
* pages). Currently the probability of moving right is set at 0.99,
* which may seem too high to change the behavior much, but it does an
* excellent job of preventing O(N^2) behavior with many equal keys.
+ *
+ * TODO: Support this old approach for pre-pg_upgrade indexes.
+ *
+ * None of this applies when all items in the tree are unique, since the
+ * new item cannot go on either page if it's equal to the high key. The
+ * original L&Y invariant that we now follow is that high keys must be
+ * less than or equal to all items on the page, and strictly less than
+ * the right sibling items (since the high key also becomes the downlink
+ * to the right sibling in parent after a page split). It's very
+ * unlikely that it will be equal anyway, since there will be explicit
+ * heap TIDs in pivot tuples in the event of many duplicates, but it can
+ * happen when heap TID recycling takes place.
*----------
*/
movedright = false;
@@ -731,8 +766,7 @@ _bt_findinsertloc(Relation rel,
* nope, so check conditions (b) and (c) enumerated above
*/
if (P_RIGHTMOST(lpageop) ||
- _bt_compare(rel, keysz, scankey, page, P_HIKEY) != 0 ||
- random() <= (MAX_RANDOM_VALUE / 100))
+ _bt_compare(rel, keysz, scankey, &newtup->t_tid, page, P_HIKEY) <= 0)
break;
/*
@@ -792,7 +826,7 @@ _bt_findinsertloc(Relation rel,
else if (firstlegaloff != InvalidOffsetNumber && !vacuumed)
newitemoff = firstlegaloff;
else
- newitemoff = _bt_binsrch(rel, buf, keysz, scankey, false);
+ newitemoff = _bt_binsrch(rel, buf, keysz, scankey, &newtup->t_tid, false);
*bufptr = buf;
*offsetptr = newitemoff;
@@ -851,11 +885,12 @@ _bt_insertonpg(Relation rel,
/* child buffer must be given iff inserting on an internal page */
Assert(P_ISLEAF(lpageop) == !BufferIsValid(cbuf));
/* tuple must have appropriate number of attributes */
+ Assert(BTreeTupleGetNAtts(itup, rel) > 0);
Assert(!P_ISLEAF(lpageop) ||
BTreeTupleGetNAtts(itup, rel) ==
IndexRelationGetNumberOfAttributes(rel));
Assert(P_ISLEAF(lpageop) ||
- BTreeTupleGetNAtts(itup, rel) ==
+ BTreeTupleGetNAtts(itup, rel) <=
IndexRelationGetNumberOfKeyAttributes(rel));
/* The caller should've finished any incomplete splits already. */
@@ -1143,8 +1178,6 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
OffsetNumber i;
bool isleaf;
IndexTuple lefthikey;
- int indnatts = IndexRelationGetNumberOfAttributes(rel);
- int indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
/* Acquire a new page to split into */
rbuf = _bt_getbuf(rel, P_NEW, BT_WRITE);
@@ -1214,7 +1247,9 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
itemid = PageGetItemId(origpage, P_HIKEY);
itemsz = ItemIdGetLength(itemid);
item = (IndexTuple) PageGetItem(origpage, itemid);
- Assert(BTreeTupleGetNAtts(item, rel) == indnkeyatts);
+ Assert(BTreeTupleGetNAtts(item, rel) > 0);
+ Assert(BTreeTupleGetNAtts(item, rel) <=
+ IndexRelationGetNumberOfKeyAttributes(rel));
if (PageAddItem(rightpage, (Item) item, itemsz, rightoff,
false, false) == InvalidOffsetNumber)
{
@@ -1247,25 +1282,35 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
}
/*
- * Truncate non-key (INCLUDE) attributes of the high key item before
- * inserting it on the left page. This only needs to happen at the leaf
- * level, since in general all pivot tuple values originate from leaf
- * level high keys. This isn't just about avoiding unnecessary work,
- * though; truncating unneeded key attributes (more aggressive suffix
- * truncation) can only be performed at the leaf level anyway. This is
- * because a pivot tuple in a grandparent page must guide a search not
- * only to the correct parent page, but also to the correct leaf page.
+ * Truncate attributes of the high key item before inserting it on the left
+ * page. This can only happen at the leaf level, since in general all
+ * pivot tuple values originate from leaf level high keys. This isn't just
+ * about avoiding unnecessary work, though; truncating unneeded key suffix
+ * attributes can only be performed at the leaf level anyway. This is
+ * because a pivot tuple in a grandparent page must guide a search not only
+ * to the correct parent page, but also to the correct leaf page.
+ *
+ * Note that non-key (INCLUDE) attributes are always truncated away here.
+ * Additional key attributes are truncated away when they're not required
+ * to correctly separate the key space.
+ *
+ * TODO: Give a little weight to how large the final downlink will be when
+ * deciding on a split point.
*/
- if (indnatts != indnkeyatts && isleaf)
+ if (isleaf)
{
- lefthikey = _bt_nonkey_truncate(rel, item);
+ OffsetNumber lastleftoffnum = OffsetNumberPrev(firstright);
+
+ lefthikey = _bt_suffix_truncate(rel, origpage, lastleftoffnum , item);
itemsz = IndexTupleSize(lefthikey);
itemsz = MAXALIGN(itemsz);
}
else
lefthikey = item;
- Assert(BTreeTupleGetNAtts(lefthikey, rel) == indnkeyatts);
+ Assert(BTreeTupleGetNAtts(lefthikey, rel) > 0);
+ Assert(BTreeTupleGetNAtts(lefthikey, rel) <=
+ IndexRelationGetNumberOfKeyAttributes(rel));
if (PageAddItem(leftpage, (Item) lefthikey, itemsz, leftoff,
false, false) == InvalidOffsetNumber)
{
@@ -1487,22 +1532,11 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
if (newitemonleft)
XLogRegisterBufData(0, (char *) newitem, MAXALIGN(newitemsz));
- /* Log left page */
- if (!isleaf || indnatts != indnkeyatts)
- {
- /*
- * We must also log the left page's high key. There are two
- * reasons for that: right page's leftmost key is suppressed on
- * non-leaf levels and in covering indexes included columns are
- * truncated from high keys. Show it as belonging to the left
- * page buffer, so that it is not stored if XLogInsert decides it
- * needs a full-page image of the left page.
- */
- itemid = PageGetItemId(origpage, P_HIKEY);
- item = (IndexTuple) PageGetItem(origpage, itemid);
- XLogRegisterBufData(0, (char *) item, MAXALIGN(IndexTupleSize(item)));
- loglhikey = true;
- }
+ /* Log left page. We must also log the left page's high key. */
+ itemid = PageGetItemId(origpage, P_HIKEY);
+ item = (IndexTuple) PageGetItem(origpage, itemid);
+ XLogRegisterBufData(0, (char *) item, MAXALIGN(IndexTupleSize(item)));
+ loglhikey = true;
/*
* Log the contents of the right page in the format understood by
@@ -2210,7 +2244,8 @@ _bt_newroot(Relation rel, Buffer lbuf, Buffer rbuf)
/*
* insert the right page pointer into the new root page.
*/
- Assert(BTreeTupleGetNAtts(right_item, rel) ==
+ Assert(BTreeTupleGetNAtts(right_item, rel) > 0);
+ Assert(BTreeTupleGetNAtts(right_item, rel) <=
IndexRelationGetNumberOfKeyAttributes(rel));
if (PageAddItem(rootpage, (Item) right_item, right_item_sz, P_FIRSTKEY,
false, false) == InvalidOffsetNumber)
@@ -2322,8 +2357,8 @@ _bt_pgaddtup(Page page,
/*
* _bt_isequal - used in _bt_doinsert in check for duplicates.
*
- * This is very similar to _bt_compare, except for NULL handling.
- * Rule is simple: NOT_NULL not equal NULL, NULL not equal NULL too.
+ * This is very similar to _bt_compare, except for NULL and negative infinity
+ * handling. Rule is simple: NOT_NULL not equal NULL, NULL not equal NULL too.
*/
static bool
_bt_isequal(TupleDesc itupdesc, Page page, OffsetNumber offnum,
@@ -2337,12 +2372,6 @@ _bt_isequal(TupleDesc itupdesc, Page page, OffsetNumber offnum,
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
- /*
- * It's okay that we might perform a comparison against a truncated page
- * high key when caller needs to determine if _bt_check_unique scan must
- * continue on to the next page. Caller never asks us to compare non-key
- * attributes within an INCLUDE index.
- */
for (i = 1; i <= keysz; i++)
{
AttrNumber attno;
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index a24e641..25b24b1 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -1415,8 +1415,10 @@ _bt_pagedel(Relation rel, Buffer buf)
itup_scankey = _bt_mkscankey(rel, targetkey);
/* find the leftmost leaf page containing this key */
stack = _bt_search(rel,
- IndexRelationGetNumberOfKeyAttributes(rel),
- itup_scankey, false, &lbuf, BT_READ, NULL);
+ BTreeTupleGetNAtts(targetkey, rel),
+ itup_scankey,
+ BTreeTupleGetHeapTID(targetkey), false,
+ &lbuf, BT_READ, NULL);
/* don't need a pin on the page */
_bt_relbuf(rel, lbuf);
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index 0bcfa10..8b70f34 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -94,8 +94,8 @@ _bt_drop_lock_and_maybe_pin(IndexScanDesc scan, BTScanPos sp)
* any incomplete splits encountered during the search will be finished.
*/
BTStack
-_bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
- Buffer *bufP, int access, Snapshot snapshot)
+_bt_search(Relation rel, int keysz, ScanKey scankey, ItemPointer scantid,
+ bool nextkey, Buffer *bufP, int access, Snapshot snapshot)
{
BTStack stack_in = NULL;
@@ -130,7 +130,7 @@ _bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
* if the leaf page is split and we insert to the parent page). But
* this is a good opportunity to finish splits of internal pages too.
*/
- *bufP = _bt_moveright(rel, *bufP, keysz, scankey, nextkey,
+ *bufP = _bt_moveright(rel, *bufP, keysz, scankey, scantid, nextkey,
(access == BT_WRITE), stack_in,
BT_READ, snapshot);
@@ -144,7 +144,7 @@ _bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
* Find the appropriate item on the internal page, and get the child
* page that it points to.
*/
- offnum = _bt_binsrch(rel, *bufP, keysz, scankey, nextkey);
+ offnum = _bt_binsrch(rel, *bufP, keysz, scankey, scantid, nextkey);
itemid = PageGetItemId(page, offnum);
itup = (IndexTuple) PageGetItem(page, itemid);
blkno = BTreeInnerTupleGetDownLink(itup);
@@ -215,6 +215,7 @@ _bt_moveright(Relation rel,
Buffer buf,
int keysz,
ScanKey scankey,
+ ItemPointer scantid,
bool nextkey,
bool forupdate,
BTStack stack,
@@ -275,7 +276,7 @@ _bt_moveright(Relation rel,
continue;
}
- if (P_IGNORE(opaque) || _bt_compare(rel, keysz, scankey, page, P_HIKEY) >= cmpval)
+ if (P_IGNORE(opaque) || _bt_compare(rel, keysz, scankey, scantid, page, P_HIKEY) >= cmpval)
{
/* step right one page */
buf = _bt_relandgetbuf(rel, buf, opaque->btpo_next, access);
@@ -324,6 +325,7 @@ _bt_binsrch(Relation rel,
Buffer buf,
int keysz,
ScanKey scankey,
+ ItemPointer scantid,
bool nextkey)
{
Page page;
@@ -371,7 +373,7 @@ _bt_binsrch(Relation rel,
/* We have low <= mid < high, so mid points at a real slot */
- result = _bt_compare(rel, keysz, scankey, page, mid);
+ result = _bt_compare(rel, keysz, scankey, scantid, page, mid);
if (result >= cmpval)
low = mid + 1;
@@ -428,24 +430,36 @@ int32
_bt_compare(Relation rel,
int keysz,
ScanKey scankey,
+ ItemPointer scantid,
Page page,
OffsetNumber offnum)
{
TupleDesc itupdesc = RelationGetDescr(rel);
BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ ItemPointer heapTid;
IndexTuple itup;
+ int ntupatts;
+ int ncmpkey;
int i;
+ Assert(keysz <= IndexRelationGetNumberOfKeyAttributes(rel));
Assert(_bt_check_natts(rel, page, offnum));
/*
* Force result ">" if target item is first data item on an internal page
* --- see NOTE above.
+ *
+ * A minus infinity key has all attributes truncated away, so this test is
+ * redundant with the minus infinity attribute tie-breaker. However, the
+ * number of attributes in minus infinity tuples was not explicitly
+ * represented as 0 until PostgreSQL v11, so an explicit offnum test is
+ * still required.
*/
if (!P_ISLEAF(opaque) && offnum == P_FIRSTDATAKEY(opaque))
return 1;
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
+ ntupatts = BTreeTupleGetNAtts(itup, rel);
/*
* The scan key is set up with the attribute number associated with each
@@ -459,7 +473,8 @@ _bt_compare(Relation rel,
* _bt_first).
*/
- for (i = 1; i <= keysz; i++)
+ ncmpkey = Min(ntupatts, keysz);
+ for (i = 1; i <= ncmpkey; i++)
{
Datum datum;
bool isNull;
@@ -510,8 +525,67 @@ _bt_compare(Relation rel,
scankey++;
}
- /* if we get here, the keys are equal */
- return 0;
+ /*
+ * Use the number of attributes as a tie-breaker, in order to treat
+ * truncated attributes in index as minus infinity.
+ */
+ if (keysz > ntupatts)
+ return 1;
+
+ /* If caller provided no heap TID tie-breaker for scan, they're equal */
+ if (!scantid)
+ return 0;
+
+ /*
+ * Although it isn't counted as an attribute by BTreeTupleGetNAtts(), heap
+ * TID is an implicit final key attribute that ensures that all index
+ * tuples have a distinct set of key attribute values.
+ *
+ * This is often truncated away in pivot tuples, which makes the attribute
+ * value implicitly negative infinity.
+ */
+ heapTid = BTreeTupleGetHeapTID(itup);
+ if (!heapTid)
+ return 1;
+ return ItemPointerCompare(scantid, heapTid);
+}
+
+/*
+ * Return how many attributes to leave when truncating.
+ *
+ * This only considers key attributes, since non-key attributes should always
+ * be truncated away. We only need attributes up to and including the first
+ * distinguishing attribute.
+ *
+ * This can return a number of attributes that is one greater than the number
+ * of key attributes actually found in the first right tuple. This indicates
+ * that the caller must use the leftmost heap TID as a unique-ifier in its new
+ * high key tuple.
+ */
+int
+_bt_leave_natts(Relation rel, Page leftpage, OffsetNumber lastleftoffnum,
+ IndexTuple firstright)
+{
+ int nkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
+ int leavenatts;
+ ScanKey skey;
+
+ skey = _bt_mkscankey(rel, firstright);
+
+ /*
+ * Even test nkeyatts (untruncated) case, since caller cares about whether
+ * or not it can avoid appending a heap TID as a unique-ifier
+ */
+ for (leavenatts = 1; leavenatts <= nkeyatts; leavenatts++)
+ {
+ if (_bt_compare(rel, leavenatts, skey, NULL, leftpage, lastleftoffnum) > 0)
+ break;
+ }
+
+ /* Can't leak memory here */
+ _bt_freeskey(skey);
+
+ return leavenatts;
}
/*
@@ -1027,7 +1101,7 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
* Use the manufactured insertion scan key to descend the tree and
* position ourselves on the target leaf page.
*/
- stack = _bt_search(rel, keysCount, scankeys, nextkey, &buf, BT_READ,
+ stack = _bt_search(rel, keysCount, scankeys, NULL, nextkey, &buf, BT_READ,
scan->xs_snapshot);
/* don't need to keep the stack around... */
@@ -1057,7 +1131,7 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
_bt_initialize_more_data(so, dir);
/* position to the precise item on the page */
- offnum = _bt_binsrch(rel, buf, keysCount, scankeys, nextkey);
+ offnum = _bt_binsrch(rel, buf, keysCount, scankeys, NULL, nextkey);
/*
* If nextkey = false, we are positioned at the first item >= scan key, or
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index e012df5..07324d4 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -805,8 +805,6 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
OffsetNumber last_off;
Size pgspc;
Size itupsz;
- int indnatts = IndexRelationGetNumberOfAttributes(wstate->index);
- int indnkeyatts = IndexRelationGetNumberOfKeyAttributes(wstate->index);
/*
* This is a handy place to check for cancel interrupts during the btree
@@ -889,17 +887,17 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
ItemIdSetUnused(ii); /* redundant */
((PageHeader) opage)->pd_lower -= sizeof(ItemIdData);
- if (indnkeyatts != indnatts && P_ISLEAF(opageop))
+ if (P_ISLEAF(opageop))
{
- IndexTuple truncated;
- Size truncsz;
+ OffsetNumber lastleftoffnum = OffsetNumberPrev(last_off);
+ IndexTuple truncated;
+ Size truncsz;
/*
- * Truncate any non-key attributes from high key on leaf level
- * (i.e. truncate on leaf level if we're building an INCLUDE
- * index). This is only done at the leaf level because downlinks
- * in internal pages are either negative infinity items, or get
- * their contents from copying from one level down. See also:
+ * Truncate away any unneeded attributes from high key on leaf
+ * level. This is only done at the leaf level because downlinks in
+ * internal pages are either negative infinity items, or get their
+ * contents from copying from one level down. See also:
* _bt_split().
*
* Since the truncated tuple is probably smaller than the
@@ -913,8 +911,12 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
* only shift the line pointer array back and forth, and overwrite
* the latter portion of the space occupied by the original tuple.
* This is fairly cheap.
+ *
+ * TODO: Give a little weight to how large the final downlink will
+ * be when deciding on a split point.
*/
- truncated = _bt_nonkey_truncate(wstate->index, oitup);
+ truncated = _bt_suffix_truncate(wstate->index, opage,
+ lastleftoffnum, oitup);
truncsz = IndexTupleSize(truncated);
PageIndexTupleDelete(opage, P_HIKEY);
_bt_sortaddtup(opage, truncsz, truncated, P_HIKEY);
@@ -933,8 +935,9 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
if (state->btps_next == NULL)
state->btps_next = _bt_pagestate(wstate, state->btps_level + 1);
- Assert(BTreeTupleGetNAtts(state->btps_minkey, wstate->index) ==
- IndexRelationGetNumberOfKeyAttributes(wstate->index) ||
+ Assert((BTreeTupleGetNAtts(state->btps_minkey, wstate->index) <=
+ IndexRelationGetNumberOfKeyAttributes(wstate->index) &&
+ BTreeTupleGetNAtts(state->btps_minkey, wstate->index) > 0) ||
P_LEFTMOST(opageop));
Assert(BTreeTupleGetNAtts(state->btps_minkey, wstate->index) == 0 ||
!P_LEFTMOST(opageop));
@@ -979,7 +982,7 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
* the first item for a page is copied from the prior page in the code
* above. Since the minimum key for an entire level is only used as a
* minus infinity downlink, and never as a high key, there is no need to
- * truncate away non-key attributes at this point.
+ * truncate away suffix attributes at this point.
*/
if (last_off == P_HIKEY)
{
@@ -1038,8 +1041,9 @@ _bt_uppershutdown(BTWriteState *wstate, BTPageState *state)
}
else
{
- Assert(BTreeTupleGetNAtts(s->btps_minkey, wstate->index) ==
- IndexRelationGetNumberOfKeyAttributes(wstate->index) ||
+ Assert((BTreeTupleGetNAtts(s->btps_minkey, wstate->index) <=
+ IndexRelationGetNumberOfKeyAttributes(wstate->index) &&
+ BTreeTupleGetNAtts(s->btps_minkey, wstate->index) > 0) ||
P_LEFTMOST(opaque));
Assert(BTreeTupleGetNAtts(s->btps_minkey, wstate->index) == 0 ||
!P_LEFTMOST(opaque));
@@ -1136,6 +1140,8 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
}
else if (itup != NULL)
{
+ int32 compare = 0;
+
for (i = 1; i <= keysz; i++)
{
SortSupport entry;
@@ -1143,7 +1149,6 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
attrDatum2;
bool isNull1,
isNull2;
- int32 compare;
entry = sortKeys + i - 1;
attrDatum1 = index_getattr(itup, i, tupdes, &isNull1);
@@ -1160,6 +1165,20 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
else if (compare < 0)
break;
}
+
+ /*
+ * If key values are equal, we sort on ItemPointer. This is
+ * required for btree indexes, since heap TID is treated as an
+ * implicit last key attribute in order to ensure that all keys
+ * in the index are physically unique.
+ */
+ if (compare == 0)
+ {
+ compare = ItemPointerCompare(&itup->t_tid, &itup2->t_tid);
+ Assert(compare != 0);
+ if (compare > 0)
+ load1 = false;
+ }
}
else
load1 = false;
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index acb9443..b9f9883 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -56,27 +56,34 @@ static bool _bt_check_rowcompare(ScanKey skey,
* Build an insertion scan key that contains comparison data from itup
* as well as comparator routines appropriate to the key datatypes.
*
- * The result is intended for use with _bt_compare().
+ * The result is intended for use with _bt_compare(). If itup has
+ * undergone suffix truncation of key attributes, caller had better
+ * pass BTreeTupleGetNAtts(itup, rel) as keysz to routines like
+ * _bt_search() and _bt_compare() when using returned scan key. This
+ * allows truncated attributes to participate in comparisons (truncated
+ * attributes have implicit negative infinity values). Note that
+ * _bt_compare() never treats a scan key as containing negative
+ * infinity attributes.
*/
ScanKey
_bt_mkscankey(Relation rel, IndexTuple itup)
{
ScanKey skey;
TupleDesc itupdesc;
+ int tupnatts;
int indnatts PG_USED_FOR_ASSERTS_ONLY;
int indnkeyatts;
int16 *indoption;
int i;
itupdesc = RelationGetDescr(rel);
+ tupnatts = BTreeTupleGetNAtts(itup, rel);
indnatts = IndexRelationGetNumberOfAttributes(rel);
indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
indoption = rel->rd_indoption;
- Assert(indnkeyatts > 0);
- Assert(indnkeyatts <= indnatts);
- Assert(BTreeTupleGetNAtts(itup, rel) == indnatts ||
- BTreeTupleGetNAtts(itup, rel) == indnkeyatts);
+ Assert(tupnatts > 0);
+ Assert(tupnatts <= indnatts);
/*
* We'll execute search using scan key constructed on key columns. Non-key
@@ -96,7 +103,21 @@ _bt_mkscankey(Relation rel, IndexTuple itup)
* comparison can be needed.
*/
procinfo = index_getprocinfo(rel, i + 1, BTORDER_PROC);
- arg = index_getattr(itup, i + 1, itupdesc, &null);
+
+ /*
+ * Truncated key attributes may not be represented in index tuple
+ * due to suffix truncation. Keys built from truncated attributes
+ * are defensively represented as NULL values, though they should
+ * still not be allowed to participate in comparisons (caller must
+ * be sure to pass a sane keysz to _bt_compare()).
+ */
+ if (i < tupnatts)
+ arg = index_getattr(itup, i + 1, itupdesc, &null);
+ else
+ {
+ arg = (Datum) 0;
+ null = true;
+ }
flags = (null ? SK_ISNULL : 0) | (indoption[i] << SK_BT_INDOPTION_SHIFT);
ScanKeyEntryInitializeWithInfo(&skey[i],
flags,
@@ -2083,38 +2104,116 @@ btproperty(Oid index_oid, int attno,
}
/*
- * _bt_nonkey_truncate() -- create tuple without non-key suffix attributes.
+ * _bt_suffix_truncate() -- create tuple without unneeded suffix attributes.
*
* Returns truncated index tuple allocated in caller's memory context, with key
- * attributes copied from caller's itup argument. Currently, suffix truncation
- * is only performed to create pivot tuples in INCLUDE indexes, but some day it
- * could be generalized to remove suffix attributes after the first
- * distinguishing key attribute.
+ * attributes copied from caller's itup argument. If rel is an INCLUDE index,
+ * non-key attributes are always truncated away, since they're not part of the
+ * key space, and are not used in pivot tuples. More aggressive suffix
+ * truncation can take place when it's clear that the returned tuple does not
+ * need one or more suffix key attributes. This is possible when there are
+ * attributes after an already distinct pair of attributes.
*
- * Truncated tuple is guaranteed to be no larger than the original, which is
- * important for staying under the 1/3 of a page restriction on tuple size.
+ * Truncated tuple is guaranteed to be no larger than the original plus space
+ * for an extra heap TID tie-breaker attribute, which is important for staying
+ * under the 1/3 of a page restriction on tuple size.
*
* Note that returned tuple's t_tid offset will hold the number of attributes
* present, so the original item pointer offset is not represented. Caller
- * should only change truncated tuple's downlink.
+ * should only change truncated tuple's downlink. Note also that truncated key
+ * attributes are treated as containing "minus infinity" values by
+ * _bt_compare().
*/
IndexTuple
-_bt_nonkey_truncate(Relation rel, IndexTuple itup)
+_bt_suffix_truncate(Relation rel, Page leftpage, OffsetNumber lastleftoffnum,
+ IndexTuple firstright)
{
- int nkeyattrs = IndexRelationGetNumberOfKeyAttributes(rel);
- IndexTuple truncated;
+ TupleDesc itupdesc = RelationGetDescr(rel);
+ int16 natts = IndexRelationGetNumberOfAttributes(rel);
+ int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
+ int leavenatts;
+ IndexTuple pivot;
+ ItemId lastleftitem;
+ IndexTuple lastlefttuple;
+ Size newsize;
+
/*
- * We should only ever truncate leaf index tuples, which must have both
- * key and non-key attributes. It's never okay to truncate a second time.
+ * We should only ever truncate leaf index tuples, which must have non-key
+ * attributes in the case of INCLUDE indexes. It's never okay to truncate
+ * a second time.
*/
- Assert(BTreeTupleGetNAtts(itup, rel) ==
- IndexRelationGetNumberOfAttributes(rel));
+ Assert(BTreeTupleGetNAtts(firstright, rel) == natts);
- truncated = index_truncate_tuple(RelationGetDescr(rel), itup, nkeyattrs);
- BTreeTupleSetNAtts(truncated, nkeyattrs);
+ /* Determine how many attributes must be left behind */
+ leavenatts = _bt_leave_natts(rel, leftpage, lastleftoffnum, firstright);
- return truncated;
+ if (leavenatts <= natts)
+ {
+ IndexTuple tidpivot;
+
+ /*
+ * Truncate away non-key attributes and/or key attributes. Do a
+ * straight copy in the case where the only attribute to be "truncated
+ * away" is the implicit heap TID key attribute (i.e. the case where we
+ * can at least avoid adding an explicit heap TID attribute to new
+ * pivot).
+ */
+ if (leavenatts < natts)
+ pivot = index_truncate_tuple(itupdesc, firstright, leavenatts);
+ else
+ pivot = CopyIndexTuple(firstright);
+
+ /*
+ * If there is a distinguishing key attribute within leavenatts, there
+ * is no need to add an explicit heap TID attribute to new pivot.
+ */
+ if (leavenatts <= nkeyatts)
+ {
+ BTreeTupleSetNAtts(pivot, leavenatts);
+ return pivot;
+ }
+
+ /*
+ * Only non-key attributes could be truncated away. They are not
+ * considered part of the key space, so it's still necessary to add a
+ * heap TID attribute to the new pivot tuple. Create enlarged copy of
+ * truncated right tuple copy, to fit heap TID.
+ */
+ newsize = IndexTupleSize(pivot) + MAXALIGN(sizeof(ItemPointerData));
+ tidpivot = palloc0(newsize);
+ memcpy(tidpivot, pivot, IndexTupleSize(pivot));
+ pfree(pivot);
+ pivot = tidpivot;
+ }
+ else
+ {
+ /*
+ * No truncation was possible. Create enlarged copy of first right
+ * tuple, to fit heap TID
+ */
+ newsize = IndexTupleSize(firstright) + MAXALIGN(sizeof(ItemPointerData));
+ pivot = palloc0(newsize);
+ memcpy(pivot, firstright, IndexTupleSize(firstright));
+ }
+
+ /*
+ * We must use heap TID as a unique-ifier in new pivot tuple, since no user
+ * key attributes could be truncated away. The heap TID must comes from
+ * the last tuple on the left page, since new downlinks must be a strict
+ * lower bound on new right page.
+ */
+ pivot->t_info &= ~INDEX_SIZE_MASK;
+ pivot->t_info |= newsize;
+ /* Copy last left item's heap TID into new pivot tuple */
+ lastleftitem = PageGetItemId(leftpage, lastleftoffnum);
+ lastlefttuple = (IndexTuple) PageGetItem(leftpage, lastleftitem);
+ memcpy((char *) pivot + newsize - MAXALIGN(sizeof(ItemPointerData)),
+ &lastlefttuple->t_tid, sizeof(ItemPointerData));
+ /* Tuple has all key attributes */
+ BTreeTupleSetNAtts(pivot, nkeyatts);
+ BTreeTupleSetHeapTID(pivot);
+ return pivot;
}
/*
@@ -2137,6 +2236,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
IndexTuple itup;
+ int tupnatts;
/*
* We cannot reliably test a deleted or half-deleted page, since they have
@@ -2156,6 +2256,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
"BT_N_KEYS_OFFSET_MASK can't fit INDEX_MAX_KEYS");
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
+ tupnatts = BTreeTupleGetNAtts(itup, rel);
if (P_ISLEAF(opaque))
{
@@ -2165,7 +2266,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
* Leaf tuples that are not the page high key (non-pivot tuples)
* should never be truncated
*/
- return BTreeTupleGetNAtts(itup, rel) == natts;
+ return tupnatts == natts;
}
else
{
@@ -2176,7 +2277,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
Assert(!P_RIGHTMOST(opaque));
/* Page high key tuple contains only key attributes */
- return BTreeTupleGetNAtts(itup, rel) == nkeyatts;
+ return tupnatts > 0 && tupnatts <= nkeyatts;
}
}
else /* !P_ISLEAF(opaque) */
@@ -2209,7 +2310,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
* Tuple contains only key attributes despite on is it page high
* key or not
*/
- return BTreeTupleGetNAtts(itup, rel) == nkeyatts;
+ return tupnatts > 0 && tupnatts <= nkeyatts;
}
}
diff --git a/src/backend/access/nbtree/nbtxlog.c b/src/backend/access/nbtree/nbtxlog.c
index 67a94cb..f1d286c 100644
--- a/src/backend/access/nbtree/nbtxlog.c
+++ b/src/backend/access/nbtree/nbtxlog.c
@@ -252,6 +252,9 @@ btree_xlog_split(bool onleft, bool lhighkey, XLogReaderState *record)
* When the high key isn't present is the wal record, then we assume it to
* be equal to the first key on the right page. It must be from the leaf
* level.
+ *
+ * FIXME: We current always log the high key. Is it worth trying to
+ * salvage case where logging isn't strictly necessary, and can be avoided?
*/
if (!lhighkey)
{
diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c
index 9fb33b9..6e8a216 100644
--- a/src/backend/utils/sort/tuplesort.c
+++ b/src/backend/utils/sort/tuplesort.c
@@ -4057,9 +4057,10 @@ comparetup_index_btree(const SortTuple *a, const SortTuple *b,
}
/*
- * If key values are equal, we sort on ItemPointer. This does not affect
- * validity of the finished index, but it may be useful to have index
- * scans in physical order.
+ * If key values are equal, we sort on ItemPointer. This is required
+ * for btree indexes, since heap TID is treated as an implicit last
+ * key attribute in order to ensure that all keys in the index are
+ * physically unique.
*/
{
BlockNumber blk1 = ItemPointerGetBlockNumber(&tuple1->t_tid);
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 3b9461c..0776324 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -122,11 +122,21 @@ typedef struct BTMetaPageData
*
* We actually need to be able to fit three items on every page,
* so restrict any one item to 1/3 the per-page available space.
+ *
+ * There are rare cases where _bt_suffix_truncate() will need to enlarge
+ * a heap index tuple to make space for a tie-breaker heap TID
+ * attribute, which we account for here.
*/
-#define BTMaxItemSize(page) \
+#define BTMaxItemSizeOld(page) \
MAXALIGN_DOWN((PageGetPageSize(page) - \
MAXALIGN(SizeOfPageHeaderData + 3*sizeof(ItemIdData)) - \
MAXALIGN(sizeof(BTPageOpaqueData))) / 3)
+#define BTMaxItemSize(page) \
+ MAXALIGN_DOWN((PageGetPageSize(page) - \
+ MAXALIGN(SizeOfPageHeaderData + \
+ 3*sizeof(ItemIdData) + \
+ 3*MAXALIGN(sizeof(ItemPointerData))) - \
+ MAXALIGN(sizeof(BTPageOpaqueData))) / 3)
/*
* The leaf-page fillfactor defaults to 90% but is user-adjustable.
@@ -204,12 +214,10 @@ typedef struct BTMetaPageData
* real offset (downlinks only need to store a block number). The offset
* field only stores the number of attributes when the INDEX_ALT_TID_MASK
* bit is set (we never assume that pivot tuples must explicitly store the
- * number of attributes, and currently do not bother storing the number of
- * attributes unless indnkeyatts actually differs from indnatts).
- * INDEX_ALT_TID_MASK is only used for pivot tuples at present, though it's
- * possible that it will be used within non-pivot tuples in the future. Do
- * not assume that a tuple with INDEX_ALT_TID_MASK set must be a pivot
- * tuple.
+ * number of attributes). INDEX_ALT_TID_MASK is only used for pivot tuples
+ * at present, though it's possible that it will be used within non-pivot
+ * tuples in the future. Do not assume that a tuple with INDEX_ALT_TID_MASK
+ * set must be a pivot tuple.
*
* The 12 least significant offset bits are used to represent the number of
* attributes in INDEX_ALT_TID_MASK tuples, leaving 4 bits that are reserved
@@ -219,6 +227,8 @@ typedef struct BTMetaPageData
#define INDEX_ALT_TID_MASK INDEX_AM_RESERVED_BIT
#define BT_RESERVED_OFFSET_MASK 0xF000
#define BT_N_KEYS_OFFSET_MASK 0x0FFF
+/* Reserved to indicate if heap TID is represented in pivot tuple */
+#define BT_HEAP_TID_ATTR 0x1000
/* Get/set downlink block number */
#define BTreeInnerTupleGetDownLink(itup) \
@@ -241,14 +251,15 @@ typedef struct BTMetaPageData
} while(0)
/*
- * Get/set number of attributes within B-tree index tuple. Asserts should be
- * removed when BT_RESERVED_OFFSET_MASK bits will be used.
+ * Get/set number of attributes within B-tree index tuple.
+ *
+ * Note that this does not include an implicit tie-breaker heap-TID
+ * attribute, if any.
*/
#define BTreeTupleGetNAtts(itup, rel) \
( \
(itup)->t_info & INDEX_ALT_TID_MASK ? \
( \
- AssertMacro((ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) & BT_RESERVED_OFFSET_MASK) == 0), \
ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) & BT_N_KEYS_OFFSET_MASK \
) \
: \
@@ -257,11 +268,33 @@ typedef struct BTMetaPageData
#define BTreeTupleSetNAtts(itup, n) \
do { \
(itup)->t_info |= INDEX_ALT_TID_MASK; \
- Assert(((n) & BT_RESERVED_OFFSET_MASK) == 0); \
ItemPointerSetOffsetNumber(&(itup)->t_tid, (n) & BT_N_KEYS_OFFSET_MASK); \
} while(0)
/*
+ * Get/set implicit tie-breaker heap-TID attribute, if any.
+ *
+ * Assumes that any tuple without INDEX_ALT_TID_MASK set has t_tid that
+ * points to heap.
+ */
+#define BTreeTupleGetHeapTID(itup) \
+ ( \
+ (itup)->t_info & INDEX_ALT_TID_MASK && \
+ (ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) & BT_HEAP_TID_ATTR) != 0 ? \
+ ( \
+ (ItemPointer) (((char *) (itup) + IndexTupleSize(itup)) - \
+ MAXALIGN(sizeof(ItemPointerData))) \
+ ) \
+ : (itup)->t_info & INDEX_ALT_TID_MASK ? NULL : (ItemPointer) &(itup)->t_tid \
+ )
+#define BTreeTupleSetHeapTID(itup) \
+ do { \
+ Assert((itup)->t_info & INDEX_ALT_TID_MASK); \
+ ItemPointerSetOffsetNumber(&(itup)->t_tid, \
+ ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) | BT_HEAP_TID_ATTR); \
+ } while(0)
+
+/*
* Operator strategy numbers for B-tree have been moved to access/stratnum.h,
* because many places need to use them in ScanKeyInit() calls.
*
@@ -564,15 +597,17 @@ extern int _bt_pagedel(Relation rel, Buffer buf);
* prototypes for functions in nbtsearch.c
*/
extern BTStack _bt_search(Relation rel,
- int keysz, ScanKey scankey, bool nextkey,
+ int keysz, ScanKey scankey, ItemPointer scantid, bool nextkey,
Buffer *bufP, int access, Snapshot snapshot);
extern Buffer _bt_moveright(Relation rel, Buffer buf, int keysz,
- ScanKey scankey, bool nextkey, bool forupdate, BTStack stack,
- int access, Snapshot snapshot);
+ ScanKey scankey, ItemPointer scantid, bool nextkey,
+ bool forupdate, BTStack stack, int access, Snapshot snapshot);
extern OffsetNumber _bt_binsrch(Relation rel, Buffer buf, int keysz,
- ScanKey scankey, bool nextkey);
+ ScanKey scankey, ItemPointer scantid, bool nextkey);
extern int32 _bt_compare(Relation rel, int keysz, ScanKey scankey,
- Page page, OffsetNumber offnum);
+ ItemPointer scantid, Page page, OffsetNumber offnum);
+extern int _bt_leave_natts(Relation rel, Page leftpage,
+ OffsetNumber lastleftoffnum, IndexTuple firstright);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
@@ -605,7 +640,9 @@ extern bytea *btoptions(Datum reloptions, bool validate);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
-extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup);
+extern IndexTuple _bt_suffix_truncate(Relation rel, Page leftpage,
+ OffsetNumber lastleftoffnum,
+ IndexTuple firstright);
extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum);
/*
diff --git a/src/test/modules/test_extensions/expected/test_extensions.out b/src/test/modules/test_extensions/expected/test_extensions.out
index 28d86c4..29b4ec9 100644
--- a/src/test/modules/test_extensions/expected/test_extensions.out
+++ b/src/test/modules/test_extensions/expected/test_extensions.out
@@ -30,10 +30,10 @@ NOTICE: installing required extension "test_ext_cyclic2"
ERROR: cyclic dependency detected between extensions "test_ext_cyclic1" and "test_ext_cyclic2"
DROP SCHEMA test_ext CASCADE;
NOTICE: drop cascades to 5 other objects
-DETAIL: drop cascades to extension test_ext3
-drop cascades to extension test_ext5
-drop cascades to extension test_ext2
+DETAIL: drop cascades to extension test_ext5
drop cascades to extension test_ext4
+drop cascades to extension test_ext3
+drop cascades to extension test_ext2
drop cascades to extension test_ext1
CREATE EXTENSION test_ext6;
DROP EXTENSION test_ext6;
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index f85e913..4f0089a 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -942,9 +942,9 @@ select distinct min(f1), max(f1) from minmaxtest;
drop table minmaxtest cascade;
NOTICE: drop cascades to 3 other objects
-DETAIL: drop cascades to table minmaxtest1
+DETAIL: drop cascades to table minmaxtest3
drop cascades to table minmaxtest2
-drop cascades to table minmaxtest3
+drop cascades to table minmaxtest1
-- check for correct detection of nested-aggregate errors
select max(min(unique1)) from tenk1;
ERROR: aggregate function calls cannot be nested
diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out
index 702bf9f..c23826f 100644
--- a/src/test/regress/expected/alter_table.out
+++ b/src/test/regress/expected/alter_table.out
@@ -2672,19 +2672,19 @@ select alter2.plus1(41);
-- clean up
drop schema alter2 cascade;
NOTICE: drop cascades to 13 other objects
-DETAIL: drop cascades to table alter2.t1
-drop cascades to view alter2.v1
-drop cascades to function alter2.plus1(integer)
-drop cascades to type alter2.posint
-drop cascades to operator family alter2.ctype_hash_ops for access method hash
+DETAIL: drop cascades to text search template alter2.tmpl
+drop cascades to text search dictionary alter2.dict
+drop cascades to text search parser alter2.prs
+drop cascades to text search configuration alter2.cfg
+drop cascades to conversion alter2.ascii_to_utf8
drop cascades to type alter2.ctype
drop cascades to function alter2.same(alter2.ctype,alter2.ctype)
drop cascades to operator alter2.=(alter2.ctype,alter2.ctype)
-drop cascades to conversion alter2.ascii_to_utf8
-drop cascades to text search parser alter2.prs
-drop cascades to text search configuration alter2.cfg
-drop cascades to text search template alter2.tmpl
-drop cascades to text search dictionary alter2.dict
+drop cascades to operator family alter2.ctype_hash_ops for access method hash
+drop cascades to type alter2.posint
+drop cascades to function alter2.plus1(integer)
+drop cascades to table alter2.t1
+drop cascades to view alter2.v1
--
-- composite types
--
diff --git a/src/test/regress/expected/collate.out b/src/test/regress/expected/collate.out
index fcbe3a5..95c7335 100644
--- a/src/test/regress/expected/collate.out
+++ b/src/test/regress/expected/collate.out
@@ -668,4 +668,4 @@ SELECT collation for ((SELECT b FROM collate_test1 LIMIT 1));
--
\set VERBOSITY terse
DROP SCHEMA collate_tests CASCADE;
-NOTICE: drop cascades to 17 other objects
+NOTICE: drop cascades to 20 other objects
diff --git a/src/test/regress/expected/create_type.out b/src/test/regress/expected/create_type.out
index 2f7d5f9..8309756 100644
--- a/src/test/regress/expected/create_type.out
+++ b/src/test/regress/expected/create_type.out
@@ -161,13 +161,13 @@ DROP FUNCTION base_fn_out(opaque); -- error
ERROR: function base_fn_out(opaque) does not exist
DROP TYPE base_type; -- error
ERROR: cannot drop type base_type because other objects depend on it
-DETAIL: function base_fn_out(base_type) depends on type base_type
-function base_fn_in(cstring) depends on type base_type
+DETAIL: function base_fn_in(cstring) depends on type base_type
+function base_fn_out(base_type) depends on type base_type
HINT: Use DROP ... CASCADE to drop the dependent objects too.
DROP TYPE base_type CASCADE;
NOTICE: drop cascades to 2 other objects
-DETAIL: drop cascades to function base_fn_out(base_type)
-drop cascades to function base_fn_in(cstring)
+DETAIL: drop cascades to function base_fn_in(cstring)
+drop cascades to function base_fn_out(base_type)
-- Check usage of typmod with a user-defined type
-- (we have borrowed numeric's typmod functions)
CREATE TEMP TABLE mytab (foo widget(42,13,7)); -- should fail
diff --git a/src/test/regress/expected/create_view.out b/src/test/regress/expected/create_view.out
index 141fc6d..8abcd7b 100644
--- a/src/test/regress/expected/create_view.out
+++ b/src/test/regress/expected/create_view.out
@@ -1711,4 +1711,4 @@ select pg_get_ruledef(oid, true) from pg_rewrite
DROP SCHEMA temp_view_test CASCADE;
NOTICE: drop cascades to 27 other objects
DROP SCHEMA testviewschm2 CASCADE;
-NOTICE: drop cascades to 62 other objects
+NOTICE: drop cascades to 63 other objects
diff --git a/src/test/regress/expected/dependency.out b/src/test/regress/expected/dependency.out
index 8e50f8f..8d31110 100644
--- a/src/test/regress/expected/dependency.out
+++ b/src/test/regress/expected/dependency.out
@@ -128,9 +128,9 @@ FROM pg_type JOIN pg_class c ON typrelid = c.oid WHERE typname = 'deptest_t';
-- doesn't work: grant still exists
DROP USER regress_dep_user1;
ERROR: role "regress_dep_user1" cannot be dropped because some objects depend on it
-DETAIL: owner of default privileges on new relations belonging to role regress_dep_user1 in schema deptest
+DETAIL: privileges for table deptest1
privileges for database regression
-privileges for table deptest1
+owner of default privileges on new relations belonging to role regress_dep_user1 in schema deptest
DROP OWNED BY regress_dep_user1;
DROP USER regress_dep_user1;
\set VERBOSITY terse
diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out
index 0b5a904..49fb047 100644
--- a/src/test/regress/expected/domain.out
+++ b/src/test/regress/expected/domain.out
@@ -645,8 +645,8 @@ alter domain dnotnulltest drop not null;
update domnotnull set col1 = null;
drop domain dnotnulltest cascade;
NOTICE: drop cascades to 2 other objects
-DETAIL: drop cascades to column col1 of table domnotnull
-drop cascades to column col2 of table domnotnull
+DETAIL: drop cascades to column col2 of table domnotnull
+drop cascades to column col1 of table domnotnull
-- Test ALTER DOMAIN .. DEFAULT ..
create table domdeftest (col1 ddef1);
insert into domdeftest default values;
diff --git a/src/test/regress/expected/event_trigger.out b/src/test/regress/expected/event_trigger.out
index 008e859..17514e6 100644
--- a/src/test/regress/expected/event_trigger.out
+++ b/src/test/regress/expected/event_trigger.out
@@ -152,9 +152,9 @@ ERROR: event trigger "regress_event_trigger" does not exist
-- should fail, regress_evt_user owns some objects
drop role regress_evt_user;
ERROR: role "regress_evt_user" cannot be dropped because some objects depend on it
-DETAIL: owner of event trigger regress_event_trigger3
+DETAIL: owner of user mapping for regress_evt_user on server useless_server
owner of default privileges on new relations belonging to role regress_evt_user
-owner of user mapping for regress_evt_user on server useless_server
+owner of event trigger regress_event_trigger3
-- cleanup before next test
-- these are all OK; the second one should emit a NOTICE
drop event trigger if exists regress_event_trigger2;
@@ -241,14 +241,13 @@ CREATE EVENT TRIGGER regress_event_trigger_drop_objects ON sql_drop
ALTER TABLE schema_one.table_one DROP COLUMN a;
DROP SCHEMA schema_one, schema_two CASCADE;
NOTICE: drop cascades to 7 other objects
-DETAIL: drop cascades to table schema_two.table_two
-drop cascades to table schema_two.table_three
-drop cascades to function schema_two.add(integer,integer)
+DETAIL: drop cascades to function schema_two.add(integer,integer)
drop cascades to function schema_two.newton(integer)
-drop cascades to table schema_one.table_one
-drop cascades to table schema_one."table two"
+drop cascades to table schema_two.table_three
+drop cascades to table schema_two.table_two
drop cascades to table schema_one.table_three
-NOTICE: table "schema_two_table_two" does not exist, skipping
+drop cascades to table schema_one."table two"
+drop cascades to table schema_one.table_one
NOTICE: table "audit_tbls_schema_two_table_three" does not exist, skipping
ERROR: object audit_tbls.schema_two_table_three of type table cannot be dropped
CONTEXT: PL/pgSQL function undroppable() line 14 at RAISE
@@ -257,61 +256,61 @@ PL/pgSQL function test_evtrig_dropped_objects() line 8 at EXECUTE
DELETE FROM undroppable_objs WHERE object_identity = 'audit_tbls.schema_two_table_three';
DROP SCHEMA schema_one, schema_two CASCADE;
NOTICE: drop cascades to 7 other objects
-DETAIL: drop cascades to table schema_two.table_two
-drop cascades to table schema_two.table_three
-drop cascades to function schema_two.add(integer,integer)
+DETAIL: drop cascades to function schema_two.add(integer,integer)
drop cascades to function schema_two.newton(integer)
-drop cascades to table schema_one.table_one
-drop cascades to table schema_one."table two"
+drop cascades to table schema_two.table_three
+drop cascades to table schema_two.table_two
drop cascades to table schema_one.table_three
-NOTICE: table "schema_two_table_two" does not exist, skipping
+drop cascades to table schema_one."table two"
+drop cascades to table schema_one.table_one
NOTICE: table "audit_tbls_schema_two_table_three" does not exist, skipping
-NOTICE: table "schema_one_table_one" does not exist, skipping
-NOTICE: table "schema_one_table two" does not exist, skipping
+NOTICE: table "schema_two_table_two" does not exist, skipping
NOTICE: table "schema_one_table_three" does not exist, skipping
+NOTICE: table "schema_one_table two" does not exist, skipping
+NOTICE: table "schema_one_table_one" does not exist, skipping
ERROR: object schema_one.table_three of type table cannot be dropped
CONTEXT: PL/pgSQL function undroppable() line 14 at RAISE
DELETE FROM undroppable_objs WHERE object_identity = 'schema_one.table_three';
DROP SCHEMA schema_one, schema_two CASCADE;
NOTICE: drop cascades to 7 other objects
-DETAIL: drop cascades to table schema_two.table_two
-drop cascades to table schema_two.table_three
-drop cascades to function schema_two.add(integer,integer)
+DETAIL: drop cascades to function schema_two.add(integer,integer)
drop cascades to function schema_two.newton(integer)
-drop cascades to table schema_one.table_one
-drop cascades to table schema_one."table two"
+drop cascades to table schema_two.table_three
+drop cascades to table schema_two.table_two
drop cascades to table schema_one.table_three
-NOTICE: table "schema_two_table_two" does not exist, skipping
+drop cascades to table schema_one."table two"
+drop cascades to table schema_one.table_one
NOTICE: table "audit_tbls_schema_two_table_three" does not exist, skipping
-NOTICE: table "schema_one_table_one" does not exist, skipping
-NOTICE: table "schema_one_table two" does not exist, skipping
+NOTICE: table "schema_two_table_two" does not exist, skipping
NOTICE: table "schema_one_table_three" does not exist, skipping
+NOTICE: table "schema_one_table two" does not exist, skipping
+NOTICE: table "schema_one_table_one" does not exist, skipping
SELECT * FROM dropped_objects WHERE schema IS NULL OR schema <> 'pg_toast';
type | schema | object
--------------+------------+-------------------------------------
table column | schema_one | schema_one.table_one.a
schema | | schema_two
- table | schema_two | schema_two.table_two
- type | schema_two | schema_two.table_two
- type | schema_two | schema_two.table_two[]
+ function | schema_two | schema_two.add(integer,integer)
+ aggregate | schema_two | schema_two.newton(integer)
table | audit_tbls | audit_tbls.schema_two_table_three
type | audit_tbls | audit_tbls.schema_two_table_three
type | audit_tbls | audit_tbls.schema_two_table_three[]
table | schema_two | schema_two.table_three
type | schema_two | schema_two.table_three
type | schema_two | schema_two.table_three[]
- function | schema_two | schema_two.add(integer,integer)
- aggregate | schema_two | schema_two.newton(integer)
+ table | schema_two | schema_two.table_two
+ type | schema_two | schema_two.table_two
+ type | schema_two | schema_two.table_two[]
schema | | schema_one
- table | schema_one | schema_one.table_one
- type | schema_one | schema_one.table_one
- type | schema_one | schema_one.table_one[]
- table | schema_one | schema_one."table two"
- type | schema_one | schema_one."table two"
- type | schema_one | schema_one."table two"[]
table | schema_one | schema_one.table_three
type | schema_one | schema_one.table_three
type | schema_one | schema_one.table_three[]
+ table | schema_one | schema_one."table two"
+ type | schema_one | schema_one."table two"
+ type | schema_one | schema_one."table two"[]
+ table | schema_one | schema_one.table_one
+ type | schema_one | schema_one.table_one
+ type | schema_one | schema_one.table_one[]
(23 rows)
DROP OWNED BY regress_evt_user;
@@ -360,13 +359,13 @@ DROP INDEX evttrig.one_idx;
NOTICE: NORMAL: orig=t normal=f istemp=f type=index identity=evttrig.one_idx name={evttrig,one_idx} args={}
DROP SCHEMA evttrig CASCADE;
NOTICE: drop cascades to 2 other objects
-DETAIL: drop cascades to table evttrig.one
-drop cascades to table evttrig.two
+DETAIL: drop cascades to table evttrig.two
+drop cascades to table evttrig.one
NOTICE: NORMAL: orig=t normal=f istemp=f type=schema identity=evttrig name={evttrig} args={}
+NOTICE: NORMAL: orig=f normal=t istemp=f type=table identity=evttrig.two name={evttrig,two} args={}
NOTICE: NORMAL: orig=f normal=t istemp=f type=table identity=evttrig.one name={evttrig,one} args={}
NOTICE: NORMAL: orig=f normal=t istemp=f type=sequence identity=evttrig.one_col_a_seq name={evttrig,one_col_a_seq} args={}
NOTICE: NORMAL: orig=f normal=t istemp=f type=default value identity=for evttrig.one.col_a name={evttrig,one,col_a} args={}
-NOTICE: NORMAL: orig=f normal=t istemp=f type=table identity=evttrig.two name={evttrig,two} args={}
DROP TABLE a_temp_tbl;
NOTICE: NORMAL: orig=t normal=f istemp=t type=table identity=pg_temp.a_temp_tbl name={pg_temp,a_temp_tbl} args={}
DROP EVENT TRIGGER regress_event_trigger_report_dropped;
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 339a43f..b5c7801 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -441,8 +441,8 @@ ALTER SERVER s1 OWNER TO regress_test_indirect;
RESET ROLE;
DROP ROLE regress_test_indirect; -- ERROR
ERROR: role "regress_test_indirect" cannot be dropped because some objects depend on it
-DETAIL: owner of server s1
-privileges for foreign-data wrapper foo
+DETAIL: privileges for foreign-data wrapper foo
+owner of server s1
\des+
List of foreign servers
Name | Owner | Foreign-data wrapper | Access privileges | Type | Version | FDW options | Description
@@ -2049,9 +2049,9 @@ DROP TABLE fd_pt2;
DROP SCHEMA foreign_schema CASCADE;
DROP ROLE regress_test_role; -- ERROR
ERROR: role "regress_test_role" cannot be dropped because some objects depend on it
-DETAIL: privileges for server s4
+DETAIL: owner of user mapping for regress_test_role on server s6
privileges for foreign-data wrapper foo
-owner of user mapping for regress_test_role on server s6
+privileges for server s4
DROP SERVER t1 CASCADE;
NOTICE: drop cascades to user mapping for public on server t1
DROP USER MAPPING FOR regress_test_role SERVER s6;
diff --git a/src/test/regress/expected/foreign_key.out b/src/test/regress/expected/foreign_key.out
index b90c492..368cc45 100644
--- a/src/test/regress/expected/foreign_key.out
+++ b/src/test/regress/expected/foreign_key.out
@@ -255,7 +255,7 @@ SELECT * FROM FKTABLE;
-- this should fail for lack of CASCADE
DROP TABLE PKTABLE;
ERROR: cannot drop table pktable because other objects depend on it
-DETAIL: constraint constrname2 on table fktable depends on table pktable
+DETAIL: constraint constrname2 on table fktable depends on index pktable_pkey
HINT: Use DROP ... CASCADE to drop the dependent objects too.
DROP TABLE PKTABLE CASCADE;
NOTICE: drop cascades to constraint constrname2 on table fktable
diff --git a/src/test/regress/expected/indexing.out b/src/test/regress/expected/indexing.out
index 2c2bf44..b3bddc0 100644
--- a/src/test/regress/expected/indexing.out
+++ b/src/test/regress/expected/indexing.out
@@ -101,8 +101,8 @@ create table idxpart (a int) partition by range (a);
create index on idxpart (a);
create table idxpart1 partition of idxpart for values from (0) to (10);
drop index idxpart1_a_idx; -- no way
-ERROR: cannot drop index idxpart1_a_idx because index idxpart_a_idx requires it
-HINT: You can drop index idxpart_a_idx instead.
+ERROR: cannot drop index idxpart1_a_idx because column a of table idxpart1 requires it
+HINT: You can drop column a of table idxpart1 instead.
drop index idxpart_a_idx; -- both indexes go away
select relname, relkind from pg_class
where relname like 'idxpart%' order by relname;
@@ -931,11 +931,11 @@ select indrelid::regclass, indexrelid::regclass, inhparent::regclass, indisvalid
(3 rows)
drop index idxpart0_pkey; -- fail
-ERROR: cannot drop index idxpart0_pkey because index idxpart_pkey requires it
-HINT: You can drop index idxpart_pkey instead.
+ERROR: cannot drop index idxpart0_pkey because constraint idxpart0_pkey on table idxpart0 requires it
+HINT: You can drop constraint idxpart0_pkey on table idxpart0 instead.
drop index idxpart1_pkey; -- fail
-ERROR: cannot drop index idxpart1_pkey because index idxpart_pkey requires it
-HINT: You can drop index idxpart_pkey instead.
+ERROR: cannot drop index idxpart1_pkey because constraint idxpart1_pkey on table idxpart1 requires it
+HINT: You can drop constraint idxpart1_pkey on table idxpart1 instead.
alter table idxpart0 drop constraint idxpart0_pkey; -- fail
ERROR: cannot drop inherited constraint "idxpart0_pkey" of relation "idxpart0"
alter table idxpart1 drop constraint idxpart1_pkey; -- fail
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
index b2b912e..3922723 100644
--- a/src/test/regress/expected/inherit.out
+++ b/src/test/regress/expected/inherit.out
@@ -952,8 +952,8 @@ Inherits: c1,
drop table p1 cascade;
NOTICE: drop cascades to 3 other objects
-DETAIL: drop cascades to table c1
-drop cascades to table c2
+DETAIL: drop cascades to table c2
+drop cascades to table c1
drop cascades to table c3
drop table p2 cascade;
create table pp1 (f1 int);
@@ -1098,9 +1098,9 @@ SELECT a.attrelid::regclass, a.attname, a.attinhcount, e.expected
DROP TABLE inht1, inhs1 CASCADE;
NOTICE: drop cascades to 4 other objects
-DETAIL: drop cascades to table inht2
+DETAIL: drop cascades to table inht3
+drop cascades to table inht2
drop cascades to table inhts
-drop cascades to table inht3
drop cascades to table inht4
-- Test non-inheritable indices [UNIQUE, EXCLUDE] constraints
CREATE TABLE test_constraints (id int, val1 varchar, val2 int, UNIQUE(val1, val2));
@@ -1352,8 +1352,8 @@ select * from patest0 join (select f1 from int4_tbl limit 1) ss on id = f1;
drop table patest0 cascade;
NOTICE: drop cascades to 2 other objects
-DETAIL: drop cascades to table patest1
-drop cascades to table patest2
+DETAIL: drop cascades to table patest2
+drop cascades to table patest1
--
-- Test merge-append plans for inheritance trees
--
@@ -1499,9 +1499,9 @@ reset enable_seqscan;
reset enable_parallel_append;
drop table matest0 cascade;
NOTICE: drop cascades to 3 other objects
-DETAIL: drop cascades to table matest1
+DETAIL: drop cascades to table matest3
drop cascades to table matest2
-drop cascades to table matest3
+drop cascades to table matest1
--
-- Check that use of an index with an extraneous column doesn't produce
-- a plan with extraneous sorting
diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out
index 08cd4be..dc78784 100644
--- a/src/test/regress/expected/matview.out
+++ b/src/test/regress/expected/matview.out
@@ -310,15 +310,15 @@ SELECT type, m.totamt AS mtot, v.totamt AS vtot FROM mvtest_tm m LEFT JOIN mvtes
-- make sure that dependencies are reported properly when they block the drop
DROP TABLE mvtest_t;
ERROR: cannot drop table mvtest_t because other objects depend on it
-DETAIL: view mvtest_tv depends on table mvtest_t
+DETAIL: materialized view mvtest_tm depends on table mvtest_t
+materialized view mvtest_tmm depends on materialized view mvtest_tm
+view mvtest_tv depends on table mvtest_t
view mvtest_tvv depends on view mvtest_tv
materialized view mvtest_tvvm depends on view mvtest_tvv
view mvtest_tvvmv depends on materialized view mvtest_tvvm
materialized view mvtest_bb depends on view mvtest_tvvmv
materialized view mvtest_mvschema.mvtest_tvm depends on view mvtest_tv
materialized view mvtest_tvmm depends on materialized view mvtest_mvschema.mvtest_tvm
-materialized view mvtest_tm depends on table mvtest_t
-materialized view mvtest_tmm depends on materialized view mvtest_tm
HINT: Use DROP ... CASCADE to drop the dependent objects too.
-- make sure dependencies are dropped and reported
-- and make sure that transactional behavior is correct on rollback
@@ -326,15 +326,15 @@ HINT: Use DROP ... CASCADE to drop the dependent objects too.
BEGIN;
DROP TABLE mvtest_t CASCADE;
NOTICE: drop cascades to 9 other objects
-DETAIL: drop cascades to view mvtest_tv
+DETAIL: drop cascades to materialized view mvtest_tm
+drop cascades to materialized view mvtest_tmm
+drop cascades to view mvtest_tv
drop cascades to view mvtest_tvv
drop cascades to materialized view mvtest_tvvm
drop cascades to view mvtest_tvvmv
drop cascades to materialized view mvtest_bb
drop cascades to materialized view mvtest_mvschema.mvtest_tvm
drop cascades to materialized view mvtest_tvmm
-drop cascades to materialized view mvtest_tm
-drop cascades to materialized view mvtest_tmm
ROLLBACK;
-- some additional tests not using base tables
CREATE VIEW mvtest_vt1 AS SELECT 1 moo;
@@ -484,10 +484,10 @@ SELECT * FROM mvtest_mv_v_4;
DROP TABLE mvtest_v CASCADE;
NOTICE: drop cascades to 4 other objects
-DETAIL: drop cascades to materialized view mvtest_mv_v
-drop cascades to materialized view mvtest_mv_v_2
+DETAIL: drop cascades to materialized view mvtest_mv_v_4
drop cascades to materialized view mvtest_mv_v_3
-drop cascades to materialized view mvtest_mv_v_4
+drop cascades to materialized view mvtest_mv_v_2
+drop cascades to materialized view mvtest_mv_v
-- Check that unknown literals are converted to "text" in CREATE MATVIEW,
-- so that we don't end up with unknown-type columns.
CREATE MATERIALIZED VIEW mv_unspecified_types AS
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index f1ae40d..8d99213 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -3502,8 +3502,8 @@ SELECT refclassid::regclass, deptype
SAVEPOINT q;
DROP ROLE regress_rls_eve; --fails due to dependency on POLICY p
ERROR: role "regress_rls_eve" cannot be dropped because some objects depend on it
-DETAIL: target of policy p on table tbl1
-privileges for table tbl1
+DETAIL: privileges for table tbl1
+target of policy p on table tbl1
ROLLBACK TO q;
ALTER POLICY p ON tbl1 TO regress_rls_frank USING (true);
SAVEPOINT q;
diff --git a/src/test/regress/expected/select_into.out b/src/test/regress/expected/select_into.out
index 942f975..30e10b1 100644
--- a/src/test/regress/expected/select_into.out
+++ b/src/test/regress/expected/select_into.out
@@ -46,9 +46,9 @@ CREATE TABLE selinto_schema.tmp3 (a,b,c)
RESET SESSION AUTHORIZATION;
DROP SCHEMA selinto_schema CASCADE;
NOTICE: drop cascades to 3 other objects
-DETAIL: drop cascades to table selinto_schema.tmp1
+DETAIL: drop cascades to table selinto_schema.tmp3
drop cascades to table selinto_schema.tmp2
-drop cascades to table selinto_schema.tmp3
+drop cascades to table selinto_schema.tmp1
DROP USER regress_selinto_user;
-- Tests for WITH NO DATA and column name consistency
CREATE TABLE ctas_base (i int, j int);
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index bf271d5..ac55ac5 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -558,9 +558,9 @@ LINE 2: FOR EACH STATEMENT WHEN (OLD.* IS DISTINCT FROM NEW.*)
-- check dependency restrictions
ALTER TABLE main_table DROP COLUMN b;
ERROR: cannot drop column b of table main_table because other objects depend on it
-DETAIL: trigger after_upd_b_row_trig on table main_table depends on column b of table main_table
+DETAIL: trigger after_upd_b_stmt_trig on table main_table depends on column b of table main_table
trigger after_upd_a_b_row_trig on table main_table depends on column b of table main_table
-trigger after_upd_b_stmt_trig on table main_table depends on column b of table main_table
+trigger after_upd_b_row_trig on table main_table depends on column b of table main_table
HINT: Use DROP ... CASCADE to drop the dependent objects too.
-- this should succeed, but we'll roll it back to keep the triggers around
begin;
@@ -1886,14 +1886,14 @@ select tgrelid::regclass, tgname, tgfoid::regproc from pg_trigger
(4 rows)
drop trigger trg1 on trigpart1; -- fail
-ERROR: cannot drop trigger trg1 on table trigpart1 because trigger trg1 on table trigpart requires it
-HINT: You can drop trigger trg1 on table trigpart instead.
+ERROR: cannot drop trigger trg1 on table trigpart1 because table trigpart1 requires it
+HINT: You can drop table trigpart1 instead.
drop trigger trg1 on trigpart2; -- fail
-ERROR: cannot drop trigger trg1 on table trigpart2 because trigger trg1 on table trigpart requires it
-HINT: You can drop trigger trg1 on table trigpart instead.
+ERROR: cannot drop trigger trg1 on table trigpart2 because table trigpart2 requires it
+HINT: You can drop table trigpart2 instead.
drop trigger trg1 on trigpart3; -- fail
-ERROR: cannot drop trigger trg1 on table trigpart3 because trigger trg1 on table trigpart requires it
-HINT: You can drop trigger trg1 on table trigpart instead.
+ERROR: cannot drop trigger trg1 on table trigpart3 because table trigpart3 requires it
+HINT: You can drop table trigpart3 instead.
drop table trigpart2; -- ok, trigger should be gone in that partition
select tgrelid::regclass, tgname, tgfoid::regproc from pg_trigger
where tgrelid::regclass::text like 'trigpart%' order by tgrelid::regclass::text;
diff --git a/src/test/regress/expected/truncate.out b/src/test/regress/expected/truncate.out
index 735d0e8..e81fa8e 100644
--- a/src/test/regress/expected/truncate.out
+++ b/src/test/regress/expected/truncate.out
@@ -278,9 +278,9 @@ SELECT * FROM trunc_faa;
ROLLBACK;
DROP TABLE trunc_f CASCADE;
NOTICE: drop cascades to 3 other objects
-DETAIL: drop cascades to table trunc_fa
+DETAIL: drop cascades to table trunc_fb
+drop cascades to table trunc_fa
drop cascades to table trunc_faa
-drop cascades to table trunc_fb
-- Test ON TRUNCATE triggers
CREATE TABLE trunc_trigger_test (f1 int, f2 text, f3 text);
CREATE TABLE trunc_trigger_log (tgop text, tglevel text, tgwhen text,
diff --git a/src/test/regress/expected/typed_table.out b/src/test/regress/expected/typed_table.out
index 2e47ecb..5834bec 100644
--- a/src/test/regress/expected/typed_table.out
+++ b/src/test/regress/expected/typed_table.out
@@ -77,17 +77,17 @@ CREATE TABLE persons4 OF person_type (
ERROR: column "name" specified more than once
DROP TYPE person_type RESTRICT;
ERROR: cannot drop type person_type because other objects depend on it
-DETAIL: table persons depends on type person_type
-function get_all_persons() depends on type person_type
+DETAIL: table persons3 depends on type person_type
table persons2 depends on type person_type
-table persons3 depends on type person_type
+function get_all_persons() depends on type person_type
+table persons depends on type person_type
HINT: Use DROP ... CASCADE to drop the dependent objects too.
DROP TYPE person_type CASCADE;
NOTICE: drop cascades to 4 other objects
-DETAIL: drop cascades to table persons
-drop cascades to function get_all_persons()
+DETAIL: drop cascades to table persons3
drop cascades to table persons2
-drop cascades to table persons3
+drop cascades to function get_all_persons()
+drop cascades to table persons
CREATE TABLE persons5 OF stuff; -- only CREATE TYPE AS types may be used
ERROR: type stuff is not a composite type
DROP TABLE stuff;
diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out
index b34bab4..1a7b3b4 100644
--- a/src/test/regress/expected/updatable_views.out
+++ b/src/test/regress/expected/updatable_views.out
@@ -328,24 +328,10 @@ UPDATE ro_view20 SET b=upper(b);
ERROR: cannot update view "ro_view20"
DETAIL: Views that return set-returning functions are not automatically updatable.
HINT: To enable updating the view, provide an INSTEAD OF UPDATE trigger or an unconditional ON UPDATE DO INSTEAD rule.
+\set VERBOSITY terse
DROP TABLE base_tbl CASCADE;
NOTICE: drop cascades to 16 other objects
-DETAIL: drop cascades to view ro_view1
-drop cascades to view ro_view17
-drop cascades to view ro_view2
-drop cascades to view ro_view3
-drop cascades to view ro_view5
-drop cascades to view ro_view6
-drop cascades to view ro_view7
-drop cascades to view ro_view8
-drop cascades to view ro_view9
-drop cascades to view ro_view11
-drop cascades to view ro_view13
-drop cascades to view rw_view15
-drop cascades to view rw_view16
-drop cascades to view ro_view20
-drop cascades to view ro_view4
-drop cascades to view rw_view14
+\set VERBOSITY default
DROP VIEW ro_view10, ro_view12, ro_view18;
DROP SEQUENCE uv_seq CASCADE;
NOTICE: drop cascades to view ro_view19
@@ -1056,8 +1042,8 @@ SELECT * FROM base_tbl;
RESET SESSION AUTHORIZATION;
DROP TABLE base_tbl CASCADE;
NOTICE: drop cascades to 2 other objects
-DETAIL: drop cascades to view rw_view1
-drop cascades to view rw_view2
+DETAIL: drop cascades to view rw_view2
+drop cascades to view rw_view1
-- nested-view permissions
CREATE TABLE base_tbl(a int, b text, c float);
INSERT INTO base_tbl VALUES (1, 'Row 1', 1.0);
@@ -1442,8 +1428,8 @@ SELECT events & 4 != 0 AS upd,
DROP TABLE base_tbl CASCADE;
NOTICE: drop cascades to 3 other objects
DETAIL: drop cascades to view rw_view1
-drop cascades to view rw_view2
drop cascades to view rw_view3
+drop cascades to view rw_view2
-- inheritance tests
CREATE TABLE base_tbl_parent (a int);
CREATE TABLE base_tbl_child (CHECK (a > 0)) INHERITS (base_tbl_parent);
@@ -1542,8 +1528,8 @@ SELECT * FROM base_tbl_child ORDER BY a;
DROP TABLE base_tbl_parent, base_tbl_child CASCADE;
NOTICE: drop cascades to 2 other objects
-DETAIL: drop cascades to view rw_view1
-drop cascades to view rw_view2
+DETAIL: drop cascades to view rw_view2
+drop cascades to view rw_view1
-- simple WITH CHECK OPTION
CREATE TABLE base_tbl (a int, b int DEFAULT 10);
INSERT INTO base_tbl VALUES (1,2), (2,3), (1,-1);
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 2443511..ddeaa72 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -242,10 +242,10 @@ NOTICE: no matching relations in tablespace "regress_tblspace_renamed" found
DROP TABLESPACE regress_tblspace_renamed;
DROP SCHEMA testschema CASCADE;
NOTICE: drop cascades to 5 other objects
-DETAIL: drop cascades to table testschema.foo
-drop cascades to table testschema.asselect
-drop cascades to table testschema.asexecute
+DETAIL: drop cascades to table testschema.tablespace_acl
drop cascades to table testschema.atable
-drop cascades to table testschema.tablespace_acl
+drop cascades to table testschema.asexecute
+drop cascades to table testschema.asselect
+drop cascades to table testschema.foo
DROP ROLE regress_tablespace_user1;
DROP ROLE regress_tablespace_user2;
diff --git a/src/test/regress/sql/updatable_views.sql b/src/test/regress/sql/updatable_views.sql
index a7786b2..deb7654 100644
--- a/src/test/regress/sql/updatable_views.sql
+++ b/src/test/regress/sql/updatable_views.sql
@@ -98,7 +98,9 @@ DELETE FROM ro_view18;
UPDATE ro_view19 SET last_value=1000;
UPDATE ro_view20 SET b=upper(b);
+\set VERBOSITY terse
DROP TABLE base_tbl CASCADE;
+\set VERBOSITY default
DROP VIEW ro_view10, ro_view12, ro_view18;
DROP SEQUENCE uv_seq CASCADE;
--
2.7.4
0004-Retail-IndexTuple-Deletion-with-TID-sorting-in-leaf-.patch (text/x-patch, 3.4K) download
From 0eca6a610245a41e36c70dd18fb11df779c28a5e Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 2 Jul 2018 16:27:53 +0500
Subject: [PATCH 4/4] Retail IndexTuple Deletion with TID sorting in leaf
nbtree pages
---
src/backend/access/nbtree/nbtree.c | 37 +++++++++++++++++++------------------
1 file changed, 19 insertions(+), 18 deletions(-)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c54aeac..041ce6f 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -887,15 +887,6 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
-static int
-tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
-{
- for (int i = 0; i < ntid; i++)
- if (ItemPointerEquals(tid, &(tid_list[i])))
- return i;
- return -1;
-}
-
IndexTargetDeleteResult*
bttargetdelete(IndexTargetDeleteInfo *info,
IndexTargetDeleteResult *stats,
@@ -914,6 +905,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
int ndeletable = 0;
OffsetNumber deletable[MaxOffsetNumber];
IndexTuple itup;
+ int pos = 0;
if (stats == NULL)
stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
@@ -922,12 +914,12 @@ bttargetdelete(IndexTargetDeleteInfo *info,
skey = _bt_mkscankey(irel, itup);
/* Descend the tree and position ourselves on the target leaf page. */
- stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
+ stack = _bt_search(irel, keysCount, skey, NULL, false, &buf, BT_READ, NULL);
_bt_freestack(stack);
/* To prepare tuple entries search across index pages */
Assert(BufferIsValid(buf));
- offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, NULL, false);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
@@ -937,7 +929,6 @@ bttargetdelete(IndexTargetDeleteInfo *info,
int32 cmpval;
ItemId itemid;
IndexTuple itup;
- int pos;
/* Switch to the next page */
if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
@@ -978,7 +969,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
/*
* This index entry satisfied to the scan key?
*/
- cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+ cmpval = _bt_compare(irel, keysCount, skey, NULL, page, offnum);
if (cmpval != 0)
/* End of index entries, satisfied to the scan key */
@@ -990,13 +981,23 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
itemid = PageGetItemId(page, offnum);
itup = (IndexTuple) PageGetItem(page, itemid);
- pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
- if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ /*
+ * Search for next TID from presorted btree result comparable
+ * to TID from presorted dead_tuples tid list
+ */
+ while (pos < info->num_dead_tuples)
{
- /* index entry for TID of dead tuple is found */
- deletable[ndeletable++] = offnum;
- info->found_dead_tuples[pos] = true;
+ int res = ItemPointerCompare(&(itup->t_tid), &info->dead_tuples[pos]);
+ if ((res == 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+ else if (res < 0)
+ break;
+ pos++;
}
offnum = OffsetNumberNext(offnum);
--
2.7.4
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-07-02 16:28 ` Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-07-02 16:28 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On Mon, Jul 2, 2018 at 7:29 AM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> In the new version the patch [1] was used in cooperation with 'retail
> indextuple deletion' and 'quick vacuum strategy' patches (see
> '0004-Retail-IndexTuple-Deletion-with-TID-sorting-in-leaf-.patch'.
Cool.
I'm going to post a revised version of the unique key patch soon. I've
found that it's slightly faster to use DESC ordering for the implicit
heap TID attribute. Doing so also restores the old user-visible
behavior for DROP dependency management, which allows me to remove all
changes to the regression test output
> Execution time of last "VACUUM test;" command on my notebook was:
>
> with bulk deletion: 1.6 s;
> with Quick Vacuum Strategy: 5.2 s;
> with Quick Vacuum Strategy & TID sorting: 0.6 s.
I'm glad that you looked into this. You could make this faster still,
by actually passing the lowest heap TID in the list of TIDs to kill to
_bt_search() and _bt_binsrch(). You might have to work through several
extra B-Tree leaf pages per bttargetdelete() call without this (you'll
move right multiple times within bttargetdelete()).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-07-02 19:40 ` Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-07-02 19:40 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On Mon, Jul 2, 2018 at 9:28 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>> Execution time of last "VACUUM test;" command on my notebook was:
>>
>> with bulk deletion: 1.6 s;
>> with Quick Vacuum Strategy: 5.2 s;
>> with Quick Vacuum Strategy & TID sorting: 0.6 s.
>
> I'm glad that you looked into this. You could make this faster still,
> by actually passing the lowest heap TID in the list of TIDs to kill to
> _bt_search() and _bt_binsrch(). You might have to work through several
> extra B-Tree leaf pages per bttargetdelete() call without this (you'll
> move right multiple times within bttargetdelete()).
I should add: I think that this doesn't matter so much in your
original test case with v1 of my patch, because you're naturally
accessing the index tuples in almost the most efficient way already,
since you VACUUM works its way from the start of the table until the
end of the table. You'll definitely need to pass a heap TID to
routines like _bt_search() once you start using my v2, though, since
that puts the heap TIDs in DESC sort order. Otherwise, it'll be almost
as slow as the plain "Quick Vacuum Strategy" case was.
In general, the big idea with my patch is that heap TID is just
another attribute. I am not "cheating" in any way; if it's not
possible to descend the tree and arrive at the right leaf page without
looking through several leaf pages, then my patch is broken.
You might also use _bt_moveright() with my patch. That way, you can
quickly detect that you need to move right immediately, without going
through all the items on the page. This should only be an issue in the
event of a concurrent page split, though. In my patch, I use
_bt_moveright() in a special way for unique indexes: I need to start
at the first leaf page a duplicate could be on for duplicate checking,
but once that's over I want to "jump immediately" to the leaf page the
index tuple actually needs to be inserted on. That's when
_bt_moveright() is called. (Actually, that looks like it breaks unique
index enforcement in the case of my patch, which I need to fix, but
you might still do this.)
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-07-03 12:17 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-07-03 12:17 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On 03.07.2018 00:40, Peter Geoghegan wrote:
> On Mon, Jul 2, 2018 at 9:28 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>>> Execution time of last "VACUUM test;" command on my notebook was:
>>>
>>> with bulk deletion: 1.6 s;
>>> with Quick Vacuum Strategy: 5.2 s;
>>> with Quick Vacuum Strategy & TID sorting: 0.6 s.
>>
>> I'm glad that you looked into this. You could make this faster still,
>> by actually passing the lowest heap TID in the list of TIDs to kill to
>> _bt_search() and _bt_binsrch(). You might have to work through several
>> extra B-Tree leaf pages per bttargetdelete() call without this (you'll
>> move right multiple times within bttargetdelete()).
>
> I should add: I think that this doesn't matter so much in your
> original test case with v1 of my patch, because you're naturally
> accessing the index tuples in almost the most efficient way already,
> since you VACUUM works its way from the start of the table until the
> end of the table. You'll definitely need to pass a heap TID to
> routines like _bt_search() once you start using my v2, though, since
> that puts the heap TIDs in DESC sort order. Otherwise, it'll be almost
> as slow as the plain "Quick Vacuum Strategy" case was.
>
> In general, the big idea with my patch is that heap TID is just
> another attribute. I am not "cheating" in any way; if it's not
> possible to descend the tree and arrive at the right leaf page without
> looking through several leaf pages, then my patch is broken.
>
> You might also use _bt_moveright() with my patch. That way, you can
> quickly detect that you need to move right immediately, without going
> through all the items on the page. This should only be an issue in the
> event of a concurrent page split, though. In my patch, I use
> _bt_moveright() in a special way for unique indexes: I need to start
> at the first leaf page a duplicate could be on for duplicate checking,
> but once that's over I want to "jump immediately" to the leaf page the
> index tuple actually needs to be inserted on. That's when
> _bt_moveright() is called. (Actually, that looks like it breaks unique
> index enforcement in the case of my patch, which I need to fix, but
> you might still do this.)
>
Done.
Attachment contains an update for use v.2 of the 'Ensure nbtree leaf
tuple keys are always unique' patch.
Apply order:
1. 0001-Retail-IndexTuple-Deletion-Access-Method.patch - from previous email
2. 0002-Quick-vacuum-strategy.patch - from previous email
3. v2-0001-Ensure-nbtree-leaf-tuple-keys-are-always-unique.patch - from [1]
4. 0004-Retail-IndexTuple-Deletion-with-TID-sorting-in-leaf.patch
[1]
https://www.postgresql.org/message-id/CAH2-Wzm6D%3DKnV%2BP8bZE-ZtP4e%2BW64HtVTdOenqd1d7HjJL3xZQ%40ma...
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
Attachments:
0004-Retail-IndexTuple-Deletion-with-TID-sorting-in-leaf.patch (text/x-patch, 6.7K) download
From 1c8569abe9479e547911ec3079633f79056eff96 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Tue, 3 Jul 2018 16:54:46 +0500
Subject: [PATCH 4/4] Retail-IndexTuple-Deletion-with-TID-sorting-in-leaf
---
src/backend/access/nbtree/nbtree.c | 75 +++++++++++++++++++++++++-------------
src/backend/commands/vacuumlazy.c | 8 ++--
src/include/access/genam.h | 2 +-
3 files changed, 55 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c54aeac..7c617e9 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -887,15 +887,19 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
-static int
-tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
-{
- for (int i = 0; i < ntid; i++)
- if (ItemPointerEquals(tid, &(tid_list[i])))
- return i;
- return -1;
-}
-
+/*
+ * Deletion of index entries pointing to heap tuples.
+ *
+ * Constraints:
+ * 1. TID list info->dead_tuples arranged in ASC order.
+ * 2. Logical duplicates of index tuples stored in DESC order.
+ *
+ * The function generates an insertion scan key and descent by btree for first
+ * index tuple what satisfies scan key and last TID in info->dead_tuples list.
+ * For the scan results it deletes all index entries, matched to the TID list.
+ *
+ * Result: a palloc'd struct containing statistical info.
+ */
IndexTargetDeleteResult*
bttargetdelete(IndexTargetDeleteInfo *info,
IndexTargetDeleteResult *stats,
@@ -914,20 +918,21 @@ bttargetdelete(IndexTargetDeleteInfo *info,
int ndeletable = 0;
OffsetNumber deletable[MaxOffsetNumber];
IndexTuple itup;
+ int pos = info->last_dead_tuple;
if (stats == NULL)
stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+ /* Assemble scankey */
itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
skey = _bt_mkscankey(irel, itup);
/* Descend the tree and position ourselves on the target leaf page. */
- stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
- _bt_freestack(stack);
+ stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_READ, NULL);
/* To prepare tuple entries search across index pages */
Assert(BufferIsValid(buf));
- offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], false);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
@@ -937,10 +942,9 @@ bttargetdelete(IndexTargetDeleteInfo *info,
int32 cmpval;
ItemId itemid;
IndexTuple itup;
- int pos;
/* Switch to the next page */
- if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ if (offnum > PageGetMaxOffsetNumber(page))
{
/*
* Before unlocking index page we need to delete
@@ -965,20 +969,27 @@ bttargetdelete(IndexTargetDeleteInfo *info,
break;
/*
- * Switch to the next index page
+ * Traverse to a next reliable index page
*/
- buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_READ, NULL);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
- offnum = P_FIRSTDATAKEY(opaque);
- continue;
+ Assert(!P_IGNORE(opaque));
+ /* Set offnum to first potentially interesting item */
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], false);
+
+ if (offnum > PageGetMaxOffsetNumber(page))
+ break;
+ else
+ continue;
}
/*
* This index entry satisfied to the scan key?
*/
- cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+ cmpval = _bt_compare(irel, keysCount, skey, NULL, page, offnum);
if (cmpval != 0)
/* End of index entries, satisfied to the scan key */
@@ -990,15 +1001,28 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
itemid = PageGetItemId(page, offnum);
itup = (IndexTuple) PageGetItem(page, itemid);
- pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
- if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ /*
+ * Search for next TID from presorted btree result comparable
+ * to TID from presorted dead_tuples tid list
+ */
+ while (pos >= 0)
{
- /* index entry for TID of dead tuple is found */
- deletable[ndeletable++] = offnum;
- info->found_dead_tuples[pos] = true;
+ int res = ItemPointerCompare(&(itup->t_tid), &info->dead_tuples[pos]);
+ if ((res == 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+ else if (res > 0)
+ break;
+ pos--;
}
+ if (pos < 0)
+ break;
+
offnum = OffsetNumberNext(offnum);
}
@@ -1015,7 +1039,8 @@ bttargetdelete(IndexTargetDeleteInfo *info,
stats->tuples_removed += ndeletable;
}
- /* Release scan key, unpin and unlock buffer */
+ /* Release stack, scan key, unpin and unlock buffer */
+ _bt_freestack(stack);
_bt_freeskey(skey);
_bt_relbuf(irel, buf);
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 0bb8abb..2fca7be 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -1770,7 +1770,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
econtext->ecxt_scantuple = slot;
/* Get tuple from heap */
- for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1814,9 +1814,9 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
- ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples - tnum;
- ivinfo.found_dead_tuples = found + tnum;
+ ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.last_dead_tuple = tnum;
+ ivinfo.found_dead_tuples = found;
index_target_delete(&ivinfo, &stats, values, isnull);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 79b386e..297d357 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -55,7 +55,7 @@ typedef struct IndexTargetDeleteInfo
{
Relation heapRelation;
Relation indexRelation; /* the index being vacuumed */
- int num_dead_tuples;
+ int last_dead_tuple;
ItemPointer dead_tuples;
bool* found_dead_tuples;
} IndexTargetDeleteInfo;
--
2.7.4
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-07-12 19:00 ` Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-23 06:25 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-23 06:47 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 3 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-07-12 19:00 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On Tue, Jul 3, 2018 at 5:17 AM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> Done.
> Attachment contains an update for use v.2 of the 'Ensure nbtree leaf tuple
> keys are always unique' patch.
My v3 is still pending, but is now a lot better than v2. There were
bugs in v2 that were fixed.
One area that might be worth investigating is retail index tuple
deletion performed within the executor in the event of non-HOT
updates. Maybe LP_REDIRECT could be repurposed to mean "ghost record",
at least in unique index tuples with no NULL values. The idea is that
MVCC index scans can skip over those if they've already found a
visible tuple with the same value. Also, when there was about to be a
page split, they could be treated a little bit like LP_DEAD items. Of
course, the ghost bit would have to be treated as a hint that could be
"wrong" (e.g. because the transaction hasn't committed yet), so you'd
have to go to the heap in the context of a page split, to double
check. Also, you'd need heuristics that let you give up on this
strategy when it didn't help.
I think that this could work well enough for OLTP workloads, and might
be more future-proof than doing it in VACUUM. Though, of course, it's
still very complicated.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-07-19 11:29 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2 siblings, 1 reply; 47+ messages in thread
From: Masahiko Sawada @ 2018-07-19 11:29 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers
On Fri, Jul 13, 2018 at 4:00 AM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Tue, Jul 3, 2018 at 5:17 AM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> Done.
>> Attachment contains an update for use v.2 of the 'Ensure nbtree leaf tuple
>> keys are always unique' patch.
>
> My v3 is still pending, but is now a lot better than v2. There were
> bugs in v2 that were fixed.
>
> One area that might be worth investigating is retail index tuple
> deletion performed within the executor in the event of non-HOT
> updates. Maybe LP_REDIRECT could be repurposed to mean "ghost record",
> at least in unique index tuples with no NULL values. The idea is that
> MVCC index scans can skip over those if they've already found a
> visible tuple with the same value.
I think that's a good idea. The overhead of marking it as ghost seems
small and it would speed up index scans. If MVCC index scans have
already found a visible tuples with the same value they can not only
skip scanning but also kill them? If can, we can kill index tuples
without checking the heap.
> Also, when there was about to be a
> page split, they could be treated a little bit like LP_DEAD items. Of
> course, the ghost bit would have to be treated as a hint that could be
> "wrong" (e.g. because the transaction hasn't committed yet), so you'd
> have to go to the heap in the context of a page split, to double
> check. Also, you'd need heuristics that let you give up on this
> strategy when it didn't help.
>
> I think that this could work well enough for OLTP workloads, and might
> be more future-proof than doing it in VACUUM. Though, of course, it's
> still very complicated.
Agreed.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-07-20 18:11 ` Peter Geoghegan <pg@bowt.ie>
2018-07-24 11:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 2 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-07-20 18:11 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers
On Thu, Jul 19, 2018 at 4:29 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> One area that might be worth investigating is retail index tuple
>> deletion performed within the executor in the event of non-HOT
>> updates. Maybe LP_REDIRECT could be repurposed to mean "ghost record",
>> at least in unique index tuples with no NULL values. The idea is that
>> MVCC index scans can skip over those if they've already found a
>> visible tuple with the same value.
>
> I think that's a good idea. The overhead of marking it as ghost seems
> small and it would speed up index scans. If MVCC index scans have
> already found a visible tuples with the same value they can not only
> skip scanning but also kill them? If can, we can kill index tuples
> without checking the heap.
I think you're talking about making LP_REDIRECT marking in nbtree
represent a "recently dead" hint: the deleting transaction has
committed, and so we are 100% sure that the tuple is about to become
garbage, but it cannot be LP_DEAD just yet because it needs to stay
around for the benefit of at least one existing snapshot/long running
transaction.
That's a different idea to what I talked about, since it could perhaps
work in a way that's much closer to LP_DEAD/kill_prior_tuple (no extra
executor stuff is required). I'm not sure if your idea is better or
worse than what I suggested, though. It would certainly be easier to
implement.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-07-24 11:38 ` Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 47+ messages in thread
From: Masahiko Sawada @ 2018-07-24 11:38 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers
On Sat, Jul 21, 2018 at 3:11 AM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Thu, Jul 19, 2018 at 4:29 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> One area that might be worth investigating is retail index tuple
>>> deletion performed within the executor in the event of non-HOT
>>> updates. Maybe LP_REDIRECT could be repurposed to mean "ghost record",
>>> at least in unique index tuples with no NULL values. The idea is that
>>> MVCC index scans can skip over those if they've already found a
>>> visible tuple with the same value.
>>
>> I think that's a good idea. The overhead of marking it as ghost seems
>> small and it would speed up index scans. If MVCC index scans have
>> already found a visible tuples with the same value they can not only
>> skip scanning but also kill them? If can, we can kill index tuples
>> without checking the heap.
>
> I think you're talking about making LP_REDIRECT marking in nbtree
> represent a "recently dead" hint: the deleting transaction has
> committed, and so we are 100% sure that the tuple is about to become
> garbage, but it cannot be LP_DEAD just yet because it needs to stay
> around for the benefit of at least one existing snapshot/long running
> transaction.
>
> That's a different idea to what I talked about, since it could perhaps
> work in a way that's much closer to LP_DEAD/kill_prior_tuple (no extra
> executor stuff is required).
I understood. Thank you for explanation!
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-08-07 07:19 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-09 00:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 2 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-08-07 07:19 UTC (permalink / raw)
To: pgsql-hackers
Hi,
I wrote a background worker (hcleaner) to demonstrate application of
Retail IndexTuple deletion (see patch at attachment).
Like Autovacuum it utilizes concept of one launcher and many workers.
But one worker correspond to one database.
Short description:
Backend collects dirty block numbers by a hash table at the point in
code immediately after heap_page_prune() call. Backend send a package of
dirty block numbers (not one-by-one!) by socket at the end of
transaction or if hash table is full.
Launcher transfers block numbers to correspond workers.
Worker collects dead tuples from a block, clean index relations, clean
heap block. It uses conditional locking with waiting list approach if
heap block are busy.
hcleaner has experimental status, but make check-world passed
.
For benchmarking i used xiaomi notebook with intel Core i5 8gen processor.
BENCHMARK
----------
test: pgbench -i -s 100 && pgbench -c 25 -j 8 -M prepared -P 60 -T 3600
autovacuum = off
master:
-------
number of transactions actually processed: 6373215
latency average = 14.122 ms
latency stddev = 9.458 ms
tps = 1770.291436 (including connections establishing)
tps = 1770.293191 (excluding connections establishing)
VACUUM verbose pgbench_accounts:
--------------------------------
INFO: vacuuming "public.pgbench_accounts"
INFO: scanned index "pgbench_accounts_pkey" to remove 237496 row versions
DETAIL: CPU: user: 4.67 s, system: 0.27 s, elapsed: 8.05 s
INFO: "pgbench_accounts": removed 237496 row versions in 167652 pages
DETAIL: CPU: user: 7.54 s, system: 3.40 s, elapsed: 26.10 s
INFO: index "pgbench_accounts_pkey" now contains 10000000 row versions
in 27422 pages
DETAIL: 237496 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: "pgbench_accounts": found 165275 removable, 10000000 nonremovable
row versions in 167840 out of 167840 pages
DETAIL: 0 dead row versions cannot be removed yet, oldest xmin: 6373796
There were 82282 unused item pointers.
Skipped 0 pages due to buffer pins, 0 frozen pages.
0 pages are entirely empty.
CPU: user: 20.33 s, system: 5.88 s, elapsed: 51.38 s.
patched:
--------
number of transactions actually processed: 6338593
latency average = 14.199 ms
latency stddev = 13.988 ms
tps = 1760.685922 (including connections establishing)
tps = 1760.688038 (excluding connections establishing)
VACUUM verbose pgbench_accounts:
--------------------------------
INFO: vacuuming "public.pgbench_accounts"
INFO: scanned index "pgbench_accounts_pkey" to remove 1804 row versions
DETAIL: CPU: user: 1.84 s, system: 0.05 s, elapsed: 3.34 s
INFO: "pgbench_accounts": removed 1804 row versions in 1468 pages
DETAIL: CPU: user: 0.06 s, system: 0.03 s, elapsed: 1.42 s
INFO: index "pgbench_accounts_pkey" now contains 10000000 row versions
in 27422 pages
DETAIL: 1618 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: "pgbench_accounts": found 168561 removable, 10000000 nonremovable
row versions in 169466 out of 169466 pages
DETAIL: 0 dead row versions cannot be removed yet, oldest xmin: 6339174
There were 75478 unused item pointers.
Skipped 0 pages due to buffer pins, 0 frozen pages.
0 pages are entirely empty.
CPU: user: 12.27 s, system: 4.03 s, elapsed: 31.43 s.
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments:
0005-Background-Cleaner.patch (text/x-patch, 42.2K) download
From 4bc8e4806b1bb1ea13dcd9e9201d392e3741adb9 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Tue, 7 Aug 2018 11:22:13 +0500
Subject: [PATCH 5/5] Background Cleaner
---
.../postgres_fdw/expected/postgres_fdw.out | 30 ++--
contrib/postgres_fdw/sql/postgres_fdw.sql | 4 +-
src/backend/access/heap/pruneheap.c | 2 +
src/backend/access/nbtree/nbtree.c | 3 +-
src/backend/access/transam/xact.c | 4 +
src/backend/commands/vacuumlazy.c | 45 +++--
src/backend/postmaster/Makefile | 2 +-
src/backend/postmaster/pgstat.c | 36 ++++
src/backend/postmaster/postmaster.c | 160 +++++++++++++++++-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/storage/lmgr/proc.c | 5 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/hash/Makefile | 2 +-
src/backend/utils/init/miscinit.c | 3 +-
src/backend/utils/init/postinit.c | 11 +-
src/include/commands/vacuum.h | 3 +
src/include/pgstat.h | 11 ++
src/include/storage/pmsignal.h | 2 +
src/test/regress/expected/rules.out | 2 +-
src/test/regress/expected/triggers.out | 2 +-
src/test/regress/input/constraints.source | 12 +-
src/test/regress/output/constraints.source | 34 ++--
src/test/regress/sql/rules.sql | 2 +-
src/test/regress/sql/triggers.sql | 2 +-
25 files changed, 317 insertions(+), 76 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index f5498c62bd..622bea2a7d 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -5496,19 +5496,25 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
- QUERY PLAN
-----------------------------------------------------------------------------------------------------------
- Update on public.ft2
- Output: c1, c2, c3, c4, c5, c6, c7, c8
- Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
- -> Foreign Scan on public.ft2
- Output: c1, c2, NULL::integer, 'bar'::text, c4, c5, c6, c7, c8, ctid
- Filter: (postgres_fdw_abs(ft2.c1) > 2000)
- Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
-(7 rows)
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------
+ Sort
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+ Sort Key: updated.c1
+ CTE updated
+ -> Update on public.ft2
+ Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: ft2.c1, ft2.c2, NULL::integer, 'bar'::text, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.ctid
+ Filter: (postgres_fdw_abs(ft2.c1) > 2000)
+ Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
+ -> CTE Scan on updated
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+(13 rows)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
------+----+-----+----+----+----+------------+----
2001 | 1 | bar | | | | ft2 |
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index e1b955f3f0..6034e83108 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1137,8 +1137,8 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c3 = 'baz'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index ebbafe3275..080bcd754d 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -22,6 +22,7 @@
#include "catalog/catalog.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
@@ -156,6 +157,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/* OK to prune */
(void) heap_page_prune(relation, buffer, OldestXmin, true, &ignore);
+ HeapCleanerSend(relation, BufferGetBlockNumber(buffer));
}
/* And release buffer lock */
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 50378e0f51..b760b30a24 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -954,6 +954,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
@@ -1033,11 +1034,11 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
_bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
-
stats->tuples_removed += ndeletable;
}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9aa63c8792..f5e70ae16d 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -40,6 +40,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -2126,6 +2127,7 @@ CommitTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(true);
AtEOXact_Snapshot(true, false);
AtEOXact_ApplyLauncher(true);
@@ -2404,6 +2406,7 @@ PrepareTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
/* don't call AtEOXact_PgStat here; we fixed pgstat state above */
AtEOXact_Snapshot(true, true);
pgstat_report_xact_timestamp(0);
@@ -2606,6 +2609,7 @@ AbortTransaction(void)
AtEOXact_Files(false);
AtEOXact_ComboCid();
AtEOXact_HashTables(false);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(false);
AtEOXact_ApplyLauncher(false);
pgstat_report_xact_timestamp(0);
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 1b952fedd9..1b6d0fc531 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -158,8 +158,6 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
-static void quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -742,7 +740,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1394,7 +1392,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1730,7 +1728,12 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
{
offnum = ItemIdGetRedirect(lp);
lp = PageGetItemId(page, offnum);
- Assert(ItemIdIsUsed(lp));
+
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
}
/* Form a tuple */
@@ -1747,30 +1750,35 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
* quick_vacuum_index() -- quick vacuum one index relation.
*
* Delete all the index entries pointing to tuples listed in
- * vacrelstats->dead_tuples.
+ * dead_tuples.
*/
-static void
+void
quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats)
+ ItemPointer dead_tuples,
+ int num_dead_tuples)
{
int tnum;
- bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
- IndexInfo* indexInfo = BuildIndexInfo(irel);
- EState* estate = CreateExecutorState();
- ExprContext* econtext = GetPerTupleExprContext(estate);
- ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
- TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ bool *found = palloc(num_dead_tuples*sizeof(bool));
+ IndexInfo *indexInfo = BuildIndexInfo(irel);
+ EState *estate = CreateExecutorState();
+ ExprContext *econtext = GetPerTupleExprContext(estate);
+ ExprState *predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
IndexTargetDeleteResult stats;
IndexTargetDeleteInfo ivinfo;
+ Assert(found != NULL);
+
+ stats.tuples_removed = 0;
ivinfo.indexRelation = irel;
ivinfo.heapRelation = hrel;
econtext->ecxt_scantuple = slot;
+ memset(found, 0, num_dead_tuples*sizeof(bool));
/* Get tuple from heap */
- for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
+ for (tnum = num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1781,7 +1789,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
continue;
/* Get a tuple from heap */
- if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ if ((tuple = get_tuple_by_tid(hrel, &(dead_tuples[tnum]))) == NULL)
{
/*
* Tuple has 'not used' status.
@@ -1814,15 +1822,16 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.dead_tuples = dead_tuples;
ivinfo.last_dead_tuple = tnum;
ivinfo.found_dead_tuples = found;
+
index_target_delete(&ivinfo, &stats, values, isnull);
}
- pfree(found);
ExecDropSingleTupleTableSlot(slot);
FreeExecutorState(estate);
+ pfree(found);
}
/*
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c23211b2..11796375e8 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
+OBJS = autovacuum.o bgheap.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index bbe73618c7..f992307e37 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -48,6 +48,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/fork_process.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
@@ -1394,6 +1395,20 @@ pgstat_report_autovac(Oid dboid)
pgstat_send(&msg, sizeof(msg));
}
+void
+pgstat_report_heapcleaner(Oid dboid)
+{
+ PgStat_MsgAutovacStart msg;
+
+ if (pgStatSock == PGINVALID_SOCKET)
+ return;
+
+ pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_BGHEAP_START);
+ msg.m_databaseid = dboid;
+ msg.m_start_time = GetCurrentTimestamp();
+
+ pgstat_send(&msg, sizeof(msg));
+}
/* ---------
* pgstat_report_vacuum() -
@@ -2833,6 +2848,14 @@ pgstat_bestart(void)
/* Autovacuum Worker */
beentry->st_backendType = B_AUTOVAC_WORKER;
}
+ else if (IsHeapCleanerLauncherProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_LAUNCHER;
+ }
+ else if (IsHeapCleanerWorkerProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_WORKER;
+ }
else if (am_walsender)
{
/* Wal sender */
@@ -3483,6 +3506,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_AUTOVACUUM_MAIN:
event_name = "AutoVacuumMain";
break;
+ case WAIT_EVENT_BGHEAP_MAIN:
+ event_name = "BgHeapMain";
+ break;
case WAIT_EVENT_BGWRITER_HIBERNATE:
event_name = "BgWriterHibernate";
break;
@@ -4111,6 +4137,12 @@ pgstat_get_backend_desc(BackendType backendType)
case B_AUTOVAC_WORKER:
backendDesc = "autovacuum worker";
break;
+ case B_BG_HEAPCLNR_LAUNCHER:
+ backendDesc = "heap cleaner launcher";
+ break;
+ case B_BG_HEAPCLNR_WORKER:
+ backendDesc = "heap cleaner worker";
+ break;
case B_BACKEND:
backendDesc = "client backend";
break;
@@ -4422,6 +4454,10 @@ PgstatCollectorMain(int argc, char *argv[])
pgstat_recv_autovac((PgStat_MsgAutovacStart *) &msg, len);
break;
+ case PGSTAT_MTYPE_BGHEAP_START:
+// pgstat_recv_autovac((PgStat_MsgHeapCleanerStart *) &msg, len);
+ break;
+
case PGSTAT_MTYPE_VACUUM:
pgstat_recv_vacuum((PgStat_MsgVacuum *) &msg, len);
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b33cd..d14c1b557d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -109,6 +109,7 @@
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
@@ -145,9 +146,10 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_BGHEAP 0x0010 /* autovacuum worker process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
-#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
+#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP)
/*
* List of active backends (or child processes anyway; we don't actually
@@ -252,6 +254,7 @@ static pid_t StartupPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
+ HeapClnrPID = 0,
PgArchPID = 0,
PgStatPID = 0,
SysLoggerPID = 0;
@@ -355,6 +358,7 @@ bool redirection_done = false; /* stderr redirected for syslogger? */
/* received START_AUTOVAC_LAUNCHER signal */
static volatile sig_atomic_t start_autovac_launcher = false;
+static volatile sig_atomic_t start_heapclnr_launcher = false;
/* the launcher needs to be signalled to communicate some condition */
static volatile bool avlauncher_needs_signal = false;
@@ -432,6 +436,7 @@ static void maybe_start_bgworkers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
+static void StartBgHeapWorker(void);
static void MaybeStartWalReceiver(void);
static void InitPostmasterDeathWatchHandle(void);
@@ -1310,6 +1315,7 @@ PostmasterMain(int argc, char *argv[])
*/
autovac_init();
+ HeapCleanerInit();
/*
* Load configuration files for client authentication.
*/
@@ -1757,6 +1763,14 @@ ServerLoop(void)
start_autovac_launcher = false; /* signal processed */
}
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && start_heapclnr_launcher &&
+ pmState == PM_RUN)
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ if (HeapClnrPID != 0)
+ start_heapclnr_launcher = false; /* signal processed */
+ }
+
/* If we have lost the stats collector, try to start a new one */
if (PgStatPID == 0 &&
(pmState == PM_RUN || pmState == PM_HOT_STANDBY))
@@ -1774,6 +1788,12 @@ ServerLoop(void)
kill(AutoVacPID, SIGUSR2);
}
+ /* If we have lost the background heap cleaner, try to start a new one. */
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && pmState == PM_RUN )
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ }
+
/* If we need to start a WAL receiver, try to do that now */
if (WalReceiverRequested)
MaybeStartWalReceiver();
@@ -2541,6 +2561,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGHUP);
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGHUP);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGHUP);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGHUP);
if (SysLoggerPID != 0)
@@ -2627,10 +2649,12 @@ pmdie(SIGNAL_ARGS)
/* autovac workers are told to shut down immediately */
/* and bgworkers too; does this need tweaking? */
SignalSomeChildren(SIGTERM,
- BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the bgwriter too */
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
@@ -2687,7 +2711,7 @@ pmdie(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGTERM);
if (pmState == PM_RECOVERY)
{
- SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER);
+ SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/*
* Only startup, bgwriter, walreceiver, possibly bgworkers,
@@ -2708,10 +2732,12 @@ pmdie(SIGNAL_ARGS)
/* shut down all backends and workers */
SignalSomeChildren(SIGTERM,
BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC |
- BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the walwriter too */
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGTERM);
@@ -2875,6 +2901,8 @@ reaper(SIGNAL_ARGS)
*/
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
+ if (!IsBinaryUpgrade && (HeapClnrPID == 0))
+ HeapClnrPID = StartHeapCleanerLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
PgArchPID = pgarch_start();
if (PgStatPID == 0)
@@ -3009,6 +3037,14 @@ reaper(SIGNAL_ARGS)
_("autovacuum launcher process"));
continue;
}
+ if (pid == HeapClnrPID)
+ {
+ HeapClnrPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("heap cleaner launcher process"));
+ continue;
+ }
/*
* Was it the archiver? If so, just try to start a new one; no need
@@ -3475,6 +3511,17 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ if (pid == HeapClnrPID)
+ HeapClnrPID = 0;
+ else if (HeapClnrPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ "SIGQUIT",
+ (int) HeapClnrPID)));
+ signal_child(HeapClnrPID, SIGQUIT);
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3661,6 +3708,7 @@ PostmasterStateMachine(void)
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
WalWriterPID == 0 &&
+ HeapClnrPID == 0 &&
AutoVacPID == 0)
{
if (Shutdown >= ImmediateShutdown || FatalError)
@@ -3759,6 +3807,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(HeapClnrPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -3948,6 +3997,8 @@ TerminateChildren(int signal)
signal_child(WalReceiverPID, signal);
if (AutoVacPID != 0)
signal_child(AutoVacPID, signal);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, signal);
if (PgArchPID != 0)
signal_child(PgArchPID, signal);
if (PgStatPID != 0)
@@ -4781,6 +4832,7 @@ SubPostmasterMain(int argc, char *argv[])
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
+ strcmp(argv[1], "--forkbgheap") == 0 ||
strcmp(argv[1], "--forkboot") == 0 ||
strncmp(argv[1], "--forkbgworker=", 15) == 0)
PGSharedMemoryReAttach();
@@ -4792,6 +4844,8 @@ SubPostmasterMain(int argc, char *argv[])
AutovacuumLauncherIAm();
if (strcmp(argv[1], "--forkavworker") == 0)
AutovacuumWorkerIAm();
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ BgHeapCleanerIAm();
/*
* Start our win32 signal implementation. This has to be done after we
@@ -4921,6 +4975,19 @@ SubPostmasterMain(int argc, char *argv[])
AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores(false, 0);
+
+ BgHeapMain(argc, argv); /* does not return */
+ }
if (strncmp(argv[1], "--forkbgworker=", 15) == 0)
{
int shmem_slot;
@@ -5122,6 +5189,12 @@ sigusr1_handler(SIGNAL_ARGS)
start_autovac_launcher = true;
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_LAUNCHER) &&
+ Shutdown == NoShutdown)
+ {
+ start_heapclnr_launcher = true;
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_AUTOVAC_WORKER) &&
Shutdown == NoShutdown)
{
@@ -5129,6 +5202,13 @@ sigusr1_handler(SIGNAL_ARGS)
StartAutovacuumWorker();
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_WORKER) &&
+ Shutdown == NoShutdown)
+ {
+ /* The autovacuum launcher wants us to start a worker process. */
+ StartBgHeapWorker();
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_WALRECEIVER))
{
/* Startup Process wants us to start the walreceiver process. */
@@ -5470,6 +5550,73 @@ StartAutovacuumWorker(void)
}
}
+static void
+StartBgHeapWorker(void)
+{
+ Backend *bn;
+
+ /*
+ * If not in condition to run a process, don't try, but handle it like a
+ * fork failure. This does not normally happen, since the signal is only
+ * supposed to be sent by autovacuum launcher when it's OK to do it, but
+ * we have to check to avoid race-condition problems during DB state
+ * changes.
+ */
+ if (canAcceptConnections() == CAC_OK)
+ {
+ /*
+ * Compute the cancel key that will be assigned to this session. We
+ * probably don't need cancel keys for autovac workers, but we'd
+ * better have something random in the field to prevent unfriendly
+ * people from sending cancels to them.
+ */
+ if (!RandomCancelKey(&MyCancelKey))
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random cancel key")));
+ return;
+ }
+
+ bn = (Backend *) malloc(sizeof(Backend));
+ if (bn)
+ {
+ bn->cancel_key = MyCancelKey;
+
+ /* Autovac workers are not dead_end and need a child slot */
+ bn->dead_end = false;
+ bn->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+ bn->bgworker_notify = false;
+
+ bn->pid = StartHeapCleanerWorker();
+
+ if (bn->pid > 0)
+ {
+ bn->bkend_type = BACKEND_TYPE_BGHEAP;
+ dlist_push_head(&BackendList, &bn->elem);
+#ifdef EXEC_BACKEND
+ ShmemBackendArrayAdd(bn);
+#endif
+ /* all OK */
+ return;
+ }
+
+ /*
+ * fork failed, fall through to report -- actual error message was
+ * logged by StartAutoVacWorker
+ */
+ (void) ReleasePostmasterChildSlot(bn->child_slot);
+ free(bn);
+ }
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+ }
+
+ elog(ERROR, "Some BGHEAP Worker start error!");
+}
+
/*
* MaybeStartWalReceiver
* Start the WAL receiver process, if not running and our state allows.
@@ -5534,7 +5681,8 @@ CreateOptsFile(int argc, char *argv[], char *fullprogname)
int
MaxLivePostmasterChildren(void)
{
- return 2 * (MaxConnections + autovacuum_max_workers + 1 +
+ return 2 * (MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 +
max_worker_processes);
}
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 0c86a581c0..fba74752a9 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -25,6 +25,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
@@ -140,6 +141,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, ProcSignalShmemSize());
size = add_size(size, CheckpointerShmemSize());
size = add_size(size, AutoVacuumShmemSize());
+ size = add_size(size, HeapCleanerShmemSize());
size = add_size(size, ReplicationSlotsShmemSize());
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
@@ -256,6 +258,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
ProcSignalShmemInit();
CheckpointerShmemInit();
AutoVacuumShmemInit();
+ HeapCleanerShmemInit();
ReplicationSlotsShmemInit();
ReplicationOriginShmemInit();
WalSndShmemInit();
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index e6025ecedb..73a9d141b0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -50,3 +50,4 @@ OldSnapshotTimeMapLock 42
BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
+HeapCleanerLock 46
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 6f30e082b2..ecf14647b4 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
@@ -352,7 +353,7 @@ InitProcess(void)
* cleaning up. (XXX autovac launcher currently doesn't participate in
* this; it probably should.)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildActive();
/*
@@ -896,7 +897,7 @@ ProcKill(int code, Datum arg)
* way, so tell the postmaster we've cleaned up acceptably well. (XXX
* autovac launcher should be included here someday)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildInactive();
/* wake autovac launcher if needed -- see comments in FreeWorkerInfo */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f4133953be..87779bd178 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -56,6 +56,7 @@
#include "parser/parser.h"
#include "pg_getopt.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -2889,6 +2890,10 @@ ProcessInterrupts(void)
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating autovacuum process due to administrator command")));
+ else if (IsHeapCleanerWorkerProcess())
+ ereport(FATAL,
+ (errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("terminating bgheap process due to administrator command")));
else if (IsLogicalWorker())
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
@@ -3017,6 +3022,13 @@ ProcessInterrupts(void)
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("canceling autovacuum task")));
}
+ if (IsHeapCleanerWorkerProcess())
+ {
+ LockErrorCleanup();
+ ereport(ERROR,
+ (errcode(ERRCODE_QUERY_CANCELED),
+ errmsg("canceling bgheap worker task")));
+ }
if (RecoveryConflictPending)
{
RecoveryConflictPending = false;
diff --git a/src/backend/utils/hash/Makefile b/src/backend/utils/hash/Makefile
index 64eebd1d99..9a3392fe37 100644
--- a/src/backend/utils/hash/Makefile
+++ b/src/backend/utils/hash/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/hash
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = dynahash.o hashfn.o pg_crc.o
+OBJS = dynahash.o hashfn.o pg_crc.o shash.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 865119d272..9ebdd1c7de 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -689,7 +690,7 @@ InitializeSessionUserIdStandalone(void)
* This function should only be called in single-user mode, in autovacuum
* workers, and in background workers.
*/
- AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker);
+ AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker || IsHeapCleanerWorkerProcess());
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 5ef6315d20..4345b8bbbc 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
@@ -321,7 +322,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
*
* We do not enforce them for autovacuum worker processes either.
*/
- if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess() && !IsHeapCleanerWorkerProcess())
{
/*
* Check that the database is currently allowing connections.
@@ -504,8 +505,8 @@ InitializeMaxBackends(void)
Assert(MaxBackends == 0);
/* the extra unit accounts for the autovacuum launcher */
- MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
- max_worker_processes;
+ MaxBackends = MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 + max_worker_processes;
/* internal error because the values were all checked previously */
if (MaxBackends > MAX_BACKENDS)
@@ -682,7 +683,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
before_shmem_exit(ShutdownPostgres, 0);
/* The autovacuum launcher is done here */
- if (IsAutoVacuumLauncherProcess())
+ if (IsAutoVacuumLauncherProcess() || IsHeapCleanerLauncherProcess())
{
/* report this backend in the PgBackendStatus array */
pgstat_bestart();
@@ -722,7 +723,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
* In standalone mode and in autovacuum worker processes, we use a fixed
* ID, otherwise we figure it out from the authenticated user name.
*/
- if (bootstrap || IsAutoVacuumWorkerProcess())
+ if (bootstrap || IsAutoVacuumWorkerProcess() || IsHeapCleanerWorkerProcess())
{
InitializeSessionUserIdStandalone();
am_superuser = true;
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 85d472f0a5..705120eddc 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -186,6 +186,9 @@ extern void vacuum_set_xid_limits(Relation rel,
extern void vac_update_datfrozenxid(void);
extern void vacuum_delay_point(void);
+extern void quick_vacuum_index(Relation irel, Relation hrel,
+ ItemPointer dead_tuples,
+ int num_dead_tuples);
/* in commands/vacuumlazy.c */
extern void lazy_vacuum_rel(Relation onerel, int options,
VacuumParams *params, BufferAccessStrategy bstrategy);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d59c24ae23..cc779063aa 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -56,6 +56,7 @@ typedef enum StatMsgType
PGSTAT_MTYPE_RESETSHAREDCOUNTER,
PGSTAT_MTYPE_RESETSINGLECOUNTER,
PGSTAT_MTYPE_AUTOVAC_START,
+ PGSTAT_MTYPE_BGHEAP_START,
PGSTAT_MTYPE_VACUUM,
PGSTAT_MTYPE_ANALYZE,
PGSTAT_MTYPE_ARCHIVER,
@@ -354,6 +355,12 @@ typedef struct PgStat_MsgAutovacStart
TimestampTz m_start_time;
} PgStat_MsgAutovacStart;
+typedef struct PgStat_MsgHeapCleanerStart
+{
+ PgStat_MsgHdr m_hdr;
+ Oid m_databaseid;
+ TimestampTz m_start_time;
+} PgStat_MsgHeapCleanerStart;
/* ----------
* PgStat_MsgVacuum Sent by the backend or autovacuum daemon
@@ -704,6 +711,8 @@ typedef enum BackendType
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
+ B_BG_HEAPCLNR_LAUNCHER,
+ B_BG_HEAPCLNR_WORKER,
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
@@ -756,6 +765,7 @@ typedef enum
{
WAIT_EVENT_ARCHIVER_MAIN = PG_WAIT_ACTIVITY,
WAIT_EVENT_AUTOVACUUM_MAIN,
+ WAIT_EVENT_BGHEAP_MAIN,
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
@@ -1181,6 +1191,7 @@ extern void pgstat_reset_shared_counters(const char *);
extern void pgstat_reset_single_counter(Oid objectid, PgStat_Single_Reset_Type type);
extern void pgstat_report_autovac(Oid dboid);
+extern void pgstat_report_heapcleaner(Oid dboid);
extern void pgstat_report_vacuum(Oid tableoid, bool shared,
PgStat_Counter livetuples, PgStat_Counter deadtuples);
extern void pgstat_report_analyze(Relation rel,
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 5ecc1b757c..4c73acd62c 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -36,6 +36,8 @@ typedef enum
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
+ PMSIGNAL_START_HEAPCLNR_LAUNCHER,
+ PMSIGNAL_START_HEAPCLNR_WORKER,
PMSIGNAL_BACKGROUND_WORKER_CHANGE, /* background worker state change */
PMSIGNAL_START_WALRECEIVER, /* start a walreceiver */
PMSIGNAL_ADVANCE_STATE_MACHINE, /* advance postmaster's state machine */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..0a0e4e8e8c 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -167,7 +167,7 @@ select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
a | b
---+----
1 | 11
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index e57c6e1c42..b9c0af7bce 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -637,7 +637,7 @@ select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
i
---
3
diff --git a/src/test/regress/input/constraints.source b/src/test/regress/input/constraints.source
index 98dd4210e9..2968a1230b 100644
--- a/src/test/regress/input/constraints.source
+++ b/src/test/regress/input/constraints.source
@@ -337,7 +337,7 @@ ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- explicitly defer the constraint
BEGIN;
@@ -349,7 +349,7 @@ DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- try adding an initially deferred constraint
ALTER TABLE unique_tbl DROP CONSTRAINT unique_tbl_i_key;
@@ -367,7 +367,7 @@ DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- should fail at commit-time
BEGIN;
@@ -420,7 +420,7 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- test a HOT update that modifies the newly inserted tuple,
-- but should succeed because we then remove the other conflicting tuple.
@@ -431,11 +431,11 @@ INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
DROP TABLE unique_tbl;
diff --git a/src/test/regress/output/constraints.source b/src/test/regress/output/constraints.source
index a6a1df18e7..a2ae6df587 100644
--- a/src/test/regress/output/constraints.source
+++ b/src/test/regress/output/constraints.source
@@ -478,7 +478,7 @@ DETAIL: Key (i)=(1) already exists.
ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+------
1 | one
@@ -494,14 +494,14 @@ SET CONSTRAINTS unique_tbl_i_key DEFERRED;
INSERT INTO unique_tbl VALUES (3, 'three');
DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
1 | one
2 | two
+ 3 | three
4 | four
5 | five
- 3 | three
(5 rows)
-- try adding an initially deferred constraint
@@ -516,14 +516,14 @@ UPDATE unique_tbl SET i = 2 WHERE i = 4 AND t = 'four';
DELETE FROM unique_tbl WHERE i = 1 AND t = 'one';
DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- should fail at commit-time
@@ -581,14 +581,14 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
ERROR: duplicate key value violates unique constraint "unique_tbl_i_key"
DETAIL: Key (i)=(3) already exists.
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- test a HOT update that modifies the newly inserted tuple,
@@ -597,25 +597,25 @@ BEGIN;
INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
DROP TABLE unique_tbl;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 3ca4c07356..c540f12a29 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -181,7 +181,7 @@ delete from rtest_v1 where b = 12;
select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
** Remember the delete rule on rtest_v1: It says
** DO INSTEAD DELETE FROM rtest_t1 WHERE a = old.a
** So this time both rows with a = 2 must get deleted
diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql
index 1250cd63e0..5a45fbad41 100644
--- a/src/test/regress/sql/triggers.sql
+++ b/src/test/regress/sql/triggers.sql
@@ -419,7 +419,7 @@ delete from trigtest where i=1;
select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
drop table trigtest2;
drop table trigtest;
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-08-09 00:19 ` Peter Geoghegan <pg@bowt.ie>
2018-08-09 03:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
1 sibling, 1 reply; 47+ messages in thread
From: Peter Geoghegan @ 2018-08-09 00:19 UTC (permalink / raw)
To: Andrey Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Tue, Aug 7, 2018 at 12:19 AM, Andrey Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> I wrote a background worker (hcleaner) to demonstrate application of Retail
> IndexTuple deletion (see patch at attachment).
> Like Autovacuum it utilizes concept of one launcher and many workers. But
> one worker correspond to one database.
>
> Short description:
> Backend collects dirty block numbers by a hash table at the point in code
> immediately after heap_page_prune() call. Backend send a package of dirty
> block numbers (not one-by-one!) by socket at the end of transaction or if
> hash table is full.
> Launcher transfers block numbers to correspond workers.
> Worker collects dead tuples from a block, clean index relations, clean heap
> block. It uses conditional locking with waiting list approach if heap block
> are busy.
>
> hcleaner has experimental status, but make check-world passed
How does this affect ordinary opportunistic pruning?
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-09 00:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-08-09 03:43 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 0 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-08-09 03:43 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-hackers
09.08.2018 05:19, Peter Geoghegan пишет:
> On Tue, Aug 7, 2018 at 12:19 AM, Andrey Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> I wrote a background worker (hcleaner) to demonstrate application of Retail
>> IndexTuple deletion (see patch at attachment).
>> Like Autovacuum it utilizes concept of one launcher and many workers. But
>> one worker correspond to one database.
>>
>> Short description:
>> Backend collects dirty block numbers by a hash table at the point in code
>> immediately after heap_page_prune() call. Backend send a package of dirty
>> block numbers (not one-by-one!) by socket at the end of transaction or if
>> hash table is full.
>> Launcher transfers block numbers to correspond workers.
>> Worker collects dead tuples from a block, clean index relations, clean heap
>> block. It uses conditional locking with waiting list approach if heap block
>> are busy.
>>
>> hcleaner has experimental status, but make check-world passed
>
> How does this affect ordinary opportunistic pruning?
>
As far as I understand, background worker not affect on ordinary
opportunistic pruning. For a dirty block it made cleaning work like
vacuum (if acquire conditional lock).
I tried two ways:
1. In executor: tid of deleted/updated tuples collects directly in
executor and send to background worker. Background worker make
heap_page_prune() himself.
But this is no good variant, because each backend perform opportunistic
pruning and it is sufficient work to detect dead tuples. Moreover, in
this case the worker compete with backends for pruning and has high load
when many backends existed.
2. In heap_page_prune_opt(): backend collects ids of dirty blocks
immediately after pruning and send it to the worker. In this case
backend perform all work for detection of dead tuples. It let the worker
to miss block cleaning during periods of heavy load: later it can clean
block.
Since locks are conditional we do not hamper backends work during high
load periods and utilize idle time for relation cleaning in soft manner.
Variant No.2 look better better and now i use it.
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-08-15 07:17 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-08-15 09:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-05 12:25 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-20 04:35 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-21 03:52 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
1 sibling, 4 replies; 47+ messages in thread
From: Masahiko Sawada @ 2018-08-15 07:17 UTC (permalink / raw)
To: Andrey Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Tue, Aug 7, 2018 at 4:19 PM, Andrey Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> Hi,
> I wrote a background worker (hcleaner) to demonstrate application of Retail
> IndexTuple deletion (see patch at attachment).
The patch doesn't seem to have the hcleaner code. Could you share it?
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-08-15 09:28 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
3 siblings, 0 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-08-15 09:28 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
15.08.2018 12:17, Masahiko Sawada пишет:
> On Tue, Aug 7, 2018 at 4:19 PM, Andrey Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> Hi,
>> I wrote a background worker (hcleaner) to demonstrate application of Retail
>> IndexTuple deletion (see patch at attachment).
>
> The patch doesn't seem to have the hcleaner code. Could you share it?
I appreciate you pointing out my mistake.
Attachment contains full patch set: indexTuple retail deletion, ordering
b-tree tuples by tid (provided by Peter Geoghean) and background cleaner.
In this version of background worker you can show state of the hcleaner
at the 'pg_stat_progress_cleaner' view, like VACUUM.
unlike the previous version, hcleaner check presence a block in memory
before cleanup (see RBM_NORMAL_NO_READ mode at ReadBufferExtended()
call) and do not read blocks from a disk storage (only on shutdown after
SIGTERM signal catch).
For feature demonstration you can use simple test (autovacuum = off):
pgbench -i -s 1 && psql -c $"CREATE INDEX pgbench_accounts_ext ON
public.pgbench_accounts USING btree (abalance);" && pgbench -t <n> -c 20
-j 8 -f test.pgb
where test.pgb is:
-------
\set aid random(1, 100000 * :scale)
\set delta random(-5000, 5000)
UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
-------
My workstation shows:
| master | patched |
n |HEAP size | INDEX size | HEAP size | INDEX size |
---------------|-------------------------------------|
2e3 | 13 MB | 2.3 MB | 13 MB | 2.3 MB |
2e4 | 14 MB | 2.7 MB | 13 MB | 2.7 MB |
2e5 | 14 MB | 8.0 MB | 14 MB | 4.8 MB |
2e6 | 61 MB | 58. MB | 14 MB | 6.7 MB |
where HEAP size - size of 'pgbench_accounts' relation; INDEX size - size
of 'pgbench_accounts_ext' index relation.
It is demonstrates a relation 'blowing' problem and influence of
hcleaner in an excessive manner.
Some problem is regression tests modification, because hcleaner makes
physical order of tuples in relations unpredictable.
>
> Regards,
>
> --
> Masahiko Sawada
> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
> NTT Open Source Software Center
>
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments: download all (5 files, .tar.gz)
0001-Retail-IndexTuple-Deletion-Access-Method.patch (text/x-patch, 12.0K) download
From 279739393a264b2d016afee3278e4afba655ce60 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 15 Aug 2018 13:34:17 +0500
Subject: [PATCH 1/5] Retail-IndexTuple-Deletion-Access-Method
---
contrib/bloom/blutils.c | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 15 +++
src/backend/access/nbtree/nbtree.c | 138 +++++++++++++++++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/include/access/amapi.h | 6 ++
src/include/access/genam.h | 18 ++++
src/include/access/nbtree.h | 4 +
11 files changed, 187 insertions(+)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3742..96f1d47c70 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbcea7..a0e06bd868 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182dd7..acf14e7bec 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42effdf7..d7a53d2ca9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df30c0..5fb32d6eba 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 22b5cc921f..9ebeb789fb 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -730,6 +730,21 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexTargetDeleteResult *
+index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation indexRelation = info->indexRelation;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, values, isnull);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index e8725fbbe1..44f4d0dc9f 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -34,7 +34,9 @@
#include "storage/smgr.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/syscache.h"
/* Working state needed by btvacuumpage */
@@ -127,6 +129,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -886,6 +889,141 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+static int
+tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
+{
+ for (int i = 0; i < ntid; i++)
+ if (ItemPointerEquals(tid, &(tid_list[i])))
+ return i;
+ return -1;
+}
+
+IndexTargetDeleteResult*
+bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation irel = info->indexRelation;
+ Relation hrel = info->heapRelation;
+ ScanKey skey;
+ int keysCount = IndexRelationGetNumberOfKeyAttributes(irel);
+ BTStack stack;
+ Buffer buf;
+ Page page;
+ BTPageOpaque opaque;
+ OffsetNumber offnum;
+ int ndeletable = 0;
+ OffsetNumber deletable[MaxOffsetNumber];
+ IndexTuple itup;
+
+ if (stats == NULL)
+ stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+
+ itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
+ skey = _bt_mkscankey(irel, itup);
+
+ /* Descend the tree and position ourselves on the target leaf page. */
+ stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
+ _bt_freestack(stack);
+
+ /* To prepare tuple entries search across index pages */
+ Assert(BufferIsValid(buf));
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ for (;;)
+ {
+ int32 cmpval;
+ ItemId itemid;
+ IndexTuple itup;
+ int pos;
+
+ /* Switch to the next page */
+ if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ {
+ /*
+ * Before unlocking index page we need to delete
+ * all currently found tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ ndeletable = 0;
+ }
+
+ /*
+ * Check for end-of-index
+ */
+ if (P_RIGHTMOST(opaque))
+ /* it is rightmost leaf */
+ break;
+
+ /*
+ * Switch to the next index page
+ */
+ buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = P_FIRSTDATAKEY(opaque);
+ continue;
+ }
+
+ /*
+ * This index entry satisfied to the scan key?
+ */
+ cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+
+ if (cmpval != 0)
+ /* End of index entries, satisfied to the scan key */
+ break;
+
+ /*
+ * To load index tuple and look for matches in the TID list of
+ * dead heap tuples
+ */
+ itemid = PageGetItemId(page, offnum);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
+
+ if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+
+ offnum = OffsetNumberNext(offnum);
+ }
+
+ /*
+ * Delete all found index tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ }
+
+ /* Release scan key, unpin and unlock buffer */
+ _bt_freeskey(skey);
+ _bt_relbuf(irel, buf);
+
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4a9b5da268..325728122b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -56,6 +56,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6bb2..497b54e8a8 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,11 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexTargetDeleteResult *(*amtargetdelete_function) (IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 24c720bf42..79b386eebd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -51,6 +51,15 @@ typedef struct IndexVacuumInfo
BufferAccessStrategy strategy; /* access strategy for reads */
} IndexVacuumInfo;
+typedef struct IndexTargetDeleteInfo
+{
+ Relation heapRelation;
+ Relation indexRelation; /* the index being vacuumed */
+ int num_dead_tuples;
+ ItemPointer dead_tuples;
+ bool* found_dead_tuples;
+} IndexTargetDeleteInfo;
+
/*
* Struct for statistics returned by ambulkdelete and amvacuumcleanup
*
@@ -79,6 +88,11 @@ typedef struct IndexBulkDeleteResult
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+typedef struct IndexTargetDeleteResult
+{
+ int tuples_removed; /* # removed during vacuum operation */
+} IndexTargetDeleteResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
@@ -163,6 +177,10 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexTargetDeleteResult *index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 04ecb4cbc0..3b9461c3b4 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -508,6 +508,10 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexTargetDeleteResult *bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
--
2.17.1
0002-Quick-Vacuum-Strategy.patch (text/x-patch, 12.3K) download
From 5cf988490d653fea24aa06e1c36af90f912fee7d Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 15 Aug 2018 13:34:45 +0500
Subject: [PATCH 2/5] Quick-Vacuum-Strategy
---
src/backend/access/heap/heapam.c | 31 ++++++
src/backend/access/heap/pruneheap.c | 40 ++++++-
src/backend/commands/vacuumlazy.c | 161 +++++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 10 +-
src/include/access/heapam.h | 1 +
5 files changed, 234 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 72395a50b8..dd033668b6 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -9392,6 +9392,36 @@ heap_sync(Relation rel)
}
}
+/*
+ * Mask DEAD tuples at a HEAP page
+ *
+ * We introduce this function for wal_consistency_checking satisfaction at
+ * Hot Standby node.
+ *
+ * DEAD tuple on a master node has t_cid and t_infomask, which can not be
+ * obtained by WAL records applying on a Hot Standby node.
+ */
+static void
+mask_dead_tuples(Page page)
+{
+ OffsetNumber offnum;
+
+ for (offnum = FirstOffsetNumber; offnum <= PageGetMaxOffsetNumber(page); offnum = OffsetNumberNext(offnum))
+ {
+ ItemId lp = PageGetItemId(page, offnum);
+ char* htup_begin;
+
+ if (!ItemIdIsDead(lp))
+ continue;
+
+ if (!ItemIdHasStorage(lp))
+ continue;
+
+ htup_begin = (char *) PageGetItem(page, lp);
+ memset(htup_begin, MASK_MARKER, ItemIdGetLength(lp));
+ }
+}
+
/*
* Mask a heap page before performing consistency checks on it.
*/
@@ -9405,6 +9435,7 @@ heap_mask(char *pagedata, BlockNumber blkno)
mask_page_hint_bits(page);
mask_unused_space(page);
+ mask_dead_tuples(page);
for (off = 1; off <= PageGetMaxOffsetNumber(page); off++)
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343dac..ebbafe3275 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -405,7 +408,7 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (HeapTupleSatisfiesVacuum(&tup, OldestXmin, buffer)
== HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
{
- heap_prune_record_unused(prstate, rootoffnum);
+ heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
ndeleted++;
@@ -580,8 +583,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
ndeleted++;
+ if (chainitems[i] == latestdead)
+ continue;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +603,28 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ /*
+ * We allow overlapping of redirected and dead items
+ */
+ heap_prune_record_redirect(prstate, rootoffnum, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
+ }
else
+ {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -653,7 +677,12 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
+
+ /*
+ * We suppress checking prstate->marked[offnum]. It is not the best idea,
+ * but this is most simplistic way to enable Dead Redirecting by
+ * overlapping Dead and Redirected states.
+ */
prstate->marked[offnum] = true;
}
@@ -706,7 +735,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 5649a70800..9a625b8dba 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -41,13 +41,16 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/nbtree.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
+#include "catalog/index.h"
#include "catalog/storage.h"
#include "commands/dbcommands.h"
#include "commands/progress.h"
#include "commands/vacuum.h"
+#include "executor/executor.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "portability/instr_time.h"
@@ -139,7 +142,6 @@ typedef struct LVRelStats
bool lock_waiter_detected;
} LVRelStats;
-
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
@@ -156,6 +158,8 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
+static void quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -734,10 +738,17 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
+ }
/*
* Report that we are now vacuuming the heap. We also increase
* the number of index scans here; note that by using
@@ -1379,10 +1390,16 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
-
+ }
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
@@ -1671,6 +1688,142 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Get tuple from heap for a scan key building.
+ */
+static HeapTuple
+get_tuple_by_tid(Relation rel, ItemPointer tid)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ ItemId lp;
+ HeapTuple tuple;
+ bool needLock = !RELATION_IS_LOCAL(rel);
+ BlockNumber npages;
+
+ if (needLock)
+ LockRelationForExtension(rel, ExclusiveLock);
+ npages = RelationGetNumberOfBlocks(rel);
+ if (needLock)
+ UnlockRelationForExtension(rel, ExclusiveLock);
+ if (ItemPointerGetBlockNumber(tid) > npages)
+ return NULL;
+
+ buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
+ LockBuffer(buffer, BUFFER_LOCK_SHARE);
+
+ page = (Page) BufferGetPage(buffer);
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
+
+ /*
+ * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
+ */
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
+ /* Walk along the chain */
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+ Assert(ItemIdIsUsed(lp));
+ }
+
+ /* Form a tuple */
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(rel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ UnlockReleaseBuffer(buffer);
+ return tuple;
+}
+
+/*
+ * quick_vacuum_index() -- quick vacuum one index relation.
+ *
+ * Delete all the index entries pointing to tuples listed in
+ * vacrelstats->dead_tuples.
+ */
+static void
+quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats)
+{
+ int tnum;
+ bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
+ IndexInfo* indexInfo = BuildIndexInfo(irel);
+ EState* estate = CreateExecutorState();
+ ExprContext* econtext = GetPerTupleExprContext(estate);
+ ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+
+ IndexTargetDeleteResult stats;
+ IndexTargetDeleteInfo ivinfo;
+
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+
+ econtext->ecxt_scantuple = slot;
+
+ /* Get tuple from heap */
+ for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ {
+ HeapTuple tuple;
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+
+ /* Index entry for the TID was deleted early */
+ if (found[tnum])
+ continue;
+
+ /* Get a tuple from heap */
+ if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ {
+ /*
+ * Tuple has 'not used' status.
+ */
+ found[tnum] = true;
+ continue;
+ }
+
+ /*
+ * Form values[] and isnull[] arrays from for index tuple
+ * by heap tuple
+ */
+ MemoryContextReset(econtext->ecxt_per_tuple_memory);
+
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * In a partial index, ignore tuples that don't satisfy the
+ * predicate.
+ */
+ if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
+ {
+ found[tnum] = true;
+ continue;
+ }
+
+ FormIndexDatum(indexInfo, slot, estate, values, isnull);
+
+ /*
+ * Make attempt to delete some index entries by one tree descent.
+ * We use only a part of TID list, which contains not found TID's.
+ */
+ ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
+ ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples - tnum;
+ ivinfo.found_dead_tuples = found + tnum;
+ index_target_delete(&ivinfo, &stats, values, isnull);
+ }
+
+ pfree(found);
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a88ea6cfc9..e9e953ece6 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3243,7 +3243,15 @@ static struct config_real ConfigureNamesReal[] =
0.1, 0.0, 100.0,
NULL, NULL, NULL
},
-
+ {
+ {"target_index_deletion_factor", PGC_SIGHUP, AUTOVACUUM,
+ gettext_noop("Maximum number of vacuumed tuples as a fraction of reltuples where we can use target index vacuum strategy."),
+ NULL
+ },
+ &target_index_deletion_factor,
+ 0.01, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
{
{"checkpoint_completion_target", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval."),
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7497..c5a7bc8361 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,7 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
/*
* HeapScanIsValid
* True iff the heap scan is valid.
--
2.17.1
0003-Make-all-nbtree-index-tuples-have-unique-keys.patch (text/x-patch, 104.1K) download
From e55cb5b5ed3507ae7e6be30f2637075eba76e42e Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 15 Aug 2018 13:35:14 +0500
Subject: [PATCH 3/5] Make-all-nbtree-index-tuples-have-unique-keys
---
contrib/amcheck/verify_nbtree.c | 256 +++++++++---
contrib/pageinspect/expected/btree.out | 2 +-
contrib/pgstattuple/expected/pgstattuple.out | 10 +-
src/backend/access/nbtree/README | 116 +++--
src/backend/access/nbtree/nbtinsert.c | 418 ++++++++++++-------
src/backend/access/nbtree/nbtpage.c | 8 +-
src/backend/access/nbtree/nbtsearch.c | 168 ++++++--
src/backend/access/nbtree/nbtsort.c | 63 ++-
src/backend/access/nbtree/nbtutils.c | 267 ++++++++++--
src/backend/access/nbtree/nbtxlog.c | 41 +-
src/backend/access/rmgrdesc/nbtdesc.c | 8 -
src/backend/storage/page/bufpage.c | 4 +-
src/backend/utils/sort/tuplesort.c | 13 +-
src/include/access/nbtree.h | 85 +++-
src/include/access/nbtxlog.h | 20 +-
src/test/regress/expected/join.out | 2 +-
16 files changed, 1064 insertions(+), 417 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index a1438a2855..eb34b22c30 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -45,6 +45,13 @@ PG_MODULE_MAGIC;
*/
#define InvalidBtreeLevel ((uint32) InvalidBlockNumber)
+/*
+ * Convenience macro to get number of key attributes in tuple in low-context
+ * fashion
+ */
+#define BTreeTupleGetNKeyAtts(itup, rel) \
+ Min(IndexRelationGetNumberOfKeyAttributes(rel), BTreeTupleGetNAtts(itup, rel))
+
/*
* State associated with verifying a B-Tree index
*
@@ -125,26 +132,30 @@ static void bt_check_every_level(Relation rel, Relation heaprel,
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_target_page_check(BtreeCheckState *state);
-static ScanKey bt_right_page_check_scankey(BtreeCheckState *state);
+static IndexTuple bt_right_page_check_tuple(BtreeCheckState *state);
static void bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
- ScanKey targetkey);
+ ScanKey targetkey, ItemPointer scantid, int tupnkeyatts);
static void bt_downlink_missing_check(BtreeCheckState *state);
static void bt_tuple_present_callback(Relation index, HeapTuple htup,
Datum *values, bool *isnull,
bool tupleIsAlive, void *checkstate);
static inline bool offset_is_negative_infinity(BTPageOpaque opaque,
OffsetNumber offset);
+static inline bool invariant_l_offset(BtreeCheckState *state,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
+ OffsetNumber upperbound);
static inline bool invariant_leq_offset(BtreeCheckState *state,
- ScanKey key,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
OffsetNumber upperbound);
-static inline bool invariant_geq_offset(BtreeCheckState *state,
- ScanKey key,
+static inline bool invariant_g_offset(BtreeCheckState *state,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
OffsetNumber lowerbound);
-static inline bool invariant_leq_nontarget_offset(BtreeCheckState *state,
- Page other,
- ScanKey key,
- OffsetNumber upperbound);
+static inline bool invariant_l_nontarget_offset(BtreeCheckState *state,
+ Page other, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber upperbound);
static Page palloc_btree_page(BtreeCheckState *state, BlockNumber blocknum);
+static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
+ IndexTuple itup, bool isleaf);
/*
* bt_index_check(index regclass, heapallindexed boolean)
@@ -834,8 +845,10 @@ bt_target_page_check(BtreeCheckState *state)
{
ItemId itemid;
IndexTuple itup;
- ScanKey skey;
size_t tupsize;
+ int tupnkeyatts;
+ ScanKey skey;
+ ItemPointer scantid;
CHECK_FOR_INTERRUPTS();
@@ -902,8 +915,17 @@ bt_target_page_check(BtreeCheckState *state)
if (offset_is_negative_infinity(topaque, offset))
continue;
- /* Build insertion scankey for current page offset */
+ /*
+ * Build insertion scankey for current page offset/tuple.
+ *
+ * As required by _bt_mkscankey(), track number of key attributes,
+ * which is needed so that _bt_compare() calls handle truncated
+ * attributes correctly. Never count non-key attributes in
+ * non-truncated tuples as key attributes, though.
+ */
+ tupnkeyatts = BTreeTupleGetNKeyAtts(itup, state->rel);
skey = _bt_mkscankey(state->rel, itup);
+ scantid = BTreeTupleGetHeapTIDCareful(state, itup, P_ISLEAF(topaque));
/* Fingerprint leaf page tuples (those that point to the heap) */
if (state->heapallindexed && P_ISLEAF(topaque) && !ItemIdIsDead(itemid))
@@ -930,7 +952,7 @@ bt_target_page_check(BtreeCheckState *state)
* and probably not markedly more effective in practice.
*/
if (!P_RIGHTMOST(topaque) &&
- !invariant_leq_offset(state, skey, P_HIKEY))
+ !invariant_leq_offset(state, tupnkeyatts, skey, scantid, P_HIKEY))
{
char *itid,
*htid;
@@ -956,11 +978,11 @@ bt_target_page_check(BtreeCheckState *state)
* * Item order check *
*
* Check that items are stored on page in logical order, by checking
- * current item is less than or equal to next item (if any).
+ * current item is strictly less than next item (if any).
*/
if (OffsetNumberNext(offset) <= max &&
- !invariant_leq_offset(state, skey,
- OffsetNumberNext(offset)))
+ !invariant_l_offset(state, tupnkeyatts, skey, scantid,
+ OffsetNumberNext(offset)))
{
char *itid,
*htid,
@@ -1017,16 +1039,29 @@ bt_target_page_check(BtreeCheckState *state)
*/
else if (offset == max)
{
+ IndexTuple righttup;
ScanKey rightkey;
+ int righttupnkeyatts;
+ ItemPointer rightscantid;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ righttup = bt_right_page_check_tuple(state);
+
+ /* Set up right item scankey */
+ if (righttup)
+ {
+ righttupnkeyatts = BTreeTupleGetNKeyAtts(righttup, state->rel);
+ rightkey = _bt_mkscankey(state->rel, righttup);
+ rightscantid = BTreeTupleGetHeapTIDCareful(state, righttup,
+ P_ISLEAF(topaque));
+ }
- if (rightkey &&
- !invariant_geq_offset(state, rightkey, max))
+ if (righttup &&
+ !invariant_g_offset(state, righttupnkeyatts, rightkey,
+ rightscantid, max))
{
/*
- * As explained at length in bt_right_page_check_scankey(),
+ * As explained at length in bt_right_page_check_tuple(),
* there is a known !readonly race that could account for
* apparent violation of invariant, which we must check for
* before actually proceeding with raising error. Our canary
@@ -1069,7 +1104,7 @@ bt_target_page_check(BtreeCheckState *state)
{
BlockNumber childblock = BTreeInnerTupleGetDownLink(itup);
- bt_downlink_check(state, childblock, skey);
+ bt_downlink_check(state, childblock, skey, scantid, tupnkeyatts);
}
}
@@ -1083,9 +1118,9 @@ bt_target_page_check(BtreeCheckState *state)
}
/*
- * Return a scankey for an item on page to right of current target (or the
+ * Return an index tuple for an item on page to right of current target (or the
* first non-ignorable page), sufficient to check ordering invariant on last
- * item in current target page. Returned scankey relies on local memory
+ * item in current target page. Returned tuple relies on local memory
* allocated for the child page, which caller cannot pfree(). Caller's memory
* context should be reset between calls here.
*
@@ -1098,8 +1133,8 @@ bt_target_page_check(BtreeCheckState *state)
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
*/
-static ScanKey
-bt_right_page_check_scankey(BtreeCheckState *state)
+static IndexTuple
+bt_right_page_check_tuple(BtreeCheckState *state)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1287,11 +1322,10 @@ bt_right_page_check_scankey(BtreeCheckState *state)
}
/*
- * Return first real item scankey. Note that this relies on right page
- * memory remaining allocated.
+ * Return first real item. Note that this relies on right page memory
+ * remaining allocated.
*/
- return _bt_mkscankey(state->rel,
- (IndexTuple) PageGetItem(rightpage, rightitem));
+ return (IndexTuple) PageGetItem(rightpage, rightitem);
}
/*
@@ -1305,7 +1339,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
*/
static void
bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
- ScanKey targetkey)
+ ScanKey targetkey, ItemPointer scantid, int tupnkeyatts)
{
OffsetNumber offset;
OffsetNumber maxoffset;
@@ -1354,7 +1388,7 @@ bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
/*
* Verify child page has the downlink key from target page (its parent) as
- * a lower bound.
+ * a lower bound; downlink must be strictly less than all keys on the page.
*
* Check all items, rather than checking just the first and trusting that
* the operator class obeys the transitive law.
@@ -1404,14 +1438,14 @@ bt_downlink_check(BtreeCheckState *state, BlockNumber childblock,
/*
* Skip comparison of target page key against "negative infinity"
* item, if any. Checking it would indicate that it's not an upper
- * bound, but that's only because of the hard-coding within
- * _bt_compare().
+ * bound, but that's only because of the hard-coding for negative
+ * inifinity items within _bt_compare().
*/
if (offset_is_negative_infinity(copaque, offset))
continue;
- if (!invariant_leq_nontarget_offset(state, child,
- targetkey, offset))
+ if (!invariant_l_nontarget_offset(state, child, tupnkeyatts, targetkey,
+ scantid, offset))
ereport(ERROR,
(errcode(ERRCODE_INDEX_CORRUPTED),
errmsg("down-link lower bound invariant violated for index \"%s\"",
@@ -1751,6 +1785,54 @@ offset_is_negative_infinity(BTPageOpaque opaque, OffsetNumber offset)
return !P_ISLEAF(opaque) && offset == P_FIRSTDATAKEY(opaque);
}
+/*
+ * Does the invariant hold that the key is strictly less than a given upper
+ * bound offset item?
+ *
+ * If this function returns false, convention is that caller throws error due
+ * to corruption.
+ */
+static inline bool
+invariant_l_offset(BtreeCheckState *state, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber upperbound)
+{
+ int32 cmp;
+
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, state->target,
+ upperbound);
+
+ /*
+ * _bt_compare interprets the absence of attributes in scan keys as meaning
+ * that they're not participating in a search, not as negative infinity
+ * (only tuples within the index are treated as negative infinity).
+ * Compensate for that here.
+ */
+ if (cmp == 0)
+ {
+ BTPageOpaque topaque;
+ ItemId itemid;
+ IndexTuple ritup;
+ int uppnkeyatts;
+ ItemPointer rheaptid;
+
+ itemid = PageGetItemId(state->target, upperbound);
+ ritup = (IndexTuple) PageGetItem(state->target, itemid);
+ uppnkeyatts = BTreeTupleGetNKeyAtts(ritup, state->rel);
+
+ /* Get heap TID for item to the right */
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+ rheaptid = BTreeTupleGetHeapTIDCareful(state, ritup,
+ P_ISLEAF(topaque));
+
+ if (uppnkeyatts == tupnkeyatts)
+ return scantid == NULL && rheaptid != NULL;
+
+ return tupnkeyatts < uppnkeyatts;
+ }
+
+ return cmp < 0;
+}
+
/*
* Does the invariant hold that the key is less than or equal to a given upper
* bound offset item?
@@ -1759,57 +1841,93 @@ offset_is_negative_infinity(BTPageOpaque opaque, OffsetNumber offset)
* to corruption.
*/
static inline bool
-invariant_leq_offset(BtreeCheckState *state, ScanKey key,
- OffsetNumber upperbound)
+invariant_leq_offset(BtreeCheckState *state, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber upperbound)
{
- int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(state->rel);
int32 cmp;
- cmp = _bt_compare(state->rel, nkeyatts, key, state->target, upperbound);
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, state->target,
+ upperbound);
return cmp <= 0;
}
/*
- * Does the invariant hold that the key is greater than or equal to a given
- * lower bound offset item?
+ * Does the invariant hold that the key is strictly greater than a given lower
+ * bound offset item?
*
* If this function returns false, convention is that caller throws error due
* to corruption.
*/
static inline bool
-invariant_geq_offset(BtreeCheckState *state, ScanKey key,
- OffsetNumber lowerbound)
+invariant_g_offset(BtreeCheckState *state, int tupnkeyatts, ScanKey key,
+ ItemPointer scantid, OffsetNumber lowerbound)
{
- int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(state->rel);
int32 cmp;
- cmp = _bt_compare(state->rel, nkeyatts, key, state->target, lowerbound);
+ /*
+ * No need to consider possibility that scankey has attributes that we need
+ * to force to be interpreted as negative infinity, since scan key has to
+ * be strictly greater than lower bound offset.
+ */
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, state->target,
+ lowerbound);
- return cmp >= 0;
+ return cmp > 0;
}
/*
- * Does the invariant hold that the key is less than or equal to a given upper
+ * Does the invariant hold that the key is strictly less than a given upper
* bound offset item, with the offset relating to a caller-supplied page that
- * is not the current target page? Caller's non-target page is typically a
- * child page of the target, checked as part of checking a property of the
- * target page (i.e. the key comes from the target).
+ * is not the current target page?
+ *
+ * Caller's non-target page is a child page of the target, checked as part of
+ * checking a property of the target page (i.e. the key comes from the
+ * target).
*
* If this function returns false, convention is that caller throws error due
* to corruption.
*/
static inline bool
-invariant_leq_nontarget_offset(BtreeCheckState *state,
- Page nontarget, ScanKey key,
- OffsetNumber upperbound)
+invariant_l_nontarget_offset(BtreeCheckState *state, Page nontarget,
+ int tupnkeyatts, ScanKey key, ItemPointer scantid,
+ OffsetNumber upperbound)
{
- int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(state->rel);
int32 cmp;
- cmp = _bt_compare(state->rel, nkeyatts, key, nontarget, upperbound);
+ cmp = _bt_compare(state->rel, tupnkeyatts, key, scantid, nontarget,
+ upperbound);
- return cmp <= 0;
+ /*
+ * _bt_compare interprets the absence of attributes in scan keys as meaning
+ * that they're not participating in a search, not as negative infinity
+ * (only tuples within the index are treated as negative infinity).
+ * Compensate for that here.
+ */
+ if (cmp == 0)
+ {
+ ItemId itemid;
+ IndexTuple child;
+ int uppnkeyatts;
+ ItemPointer childheaptid;
+ BTPageOpaque copaque;
+
+ copaque = (BTPageOpaque) PageGetSpecialPointer(nontarget);
+ itemid = PageGetItemId(nontarget, upperbound);
+ child = (IndexTuple) PageGetItem(nontarget, itemid);
+ uppnkeyatts = BTreeTupleGetNKeyAtts(child, state->rel);
+
+ /* Get heap TID for item from child/non-target */
+ childheaptid = BTreeTupleGetHeapTIDCareful(state, child,
+ P_ISLEAF(copaque));
+
+ if (uppnkeyatts == tupnkeyatts)
+ return scantid == NULL && childheaptid != NULL;
+
+ return tupnkeyatts < uppnkeyatts;
+ }
+
+ return cmp < 0;
}
/*
@@ -1965,3 +2083,31 @@ palloc_btree_page(BtreeCheckState *state, BlockNumber blocknum)
return page;
}
+
+/*
+ * BTreeTupleGetHeapTID() wrapper that lets caller enforce that a heap TID must
+ * be present in cases where that is mandatory.
+ *
+ * This doesn't add much as of BTREE_VERSION 4, since the INDEX_ALT_TID_MASK
+ * bit is effectively a proxy for whether or not the tuple is a pivot tuple.
+ * It may become more useful in the future, when non-pivot tuples support their
+ * own alternative INDEX_ALT_TID_MASK representation.
+ *
+ * Note that it is incorrect to specify the tuple as a non-pivot when passing a
+ * leaf tuple that came from the high key offset, since that is actually a
+ * pivot tuple.
+ */
+static inline ItemPointer
+BTreeTupleGetHeapTIDCareful(BtreeCheckState *state, IndexTuple itup,
+ bool nonpivot)
+{
+ ItemPointer result = BTreeTupleGetHeapTID(itup);
+
+ if (result == NULL && nonpivot)
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("block %u or its right sibling block or child block in index \"%s\" contains non-pivot tuple that lacks a heap TID",
+ state->targetblock, RelationGetRelationName(state->rel))));
+
+ return result;
+}
diff --git a/contrib/pageinspect/expected/btree.out b/contrib/pageinspect/expected/btree.out
index 2aaa4df53b..07c2dcd771 100644
--- a/contrib/pageinspect/expected/btree.out
+++ b/contrib/pageinspect/expected/btree.out
@@ -5,7 +5,7 @@ CREATE INDEX test1_a_idx ON test1 USING btree (a);
SELECT * FROM bt_metap('test1_a_idx');
-[ RECORD 1 ]-----------+-------
magic | 340322
-version | 3
+version | 4
root | 1
level | 0
fastroot | 1
diff --git a/contrib/pgstattuple/expected/pgstattuple.out b/contrib/pgstattuple/expected/pgstattuple.out
index 9858ea69d4..9920dbfd40 100644
--- a/contrib/pgstattuple/expected/pgstattuple.out
+++ b/contrib/pgstattuple/expected/pgstattuple.out
@@ -48,7 +48,7 @@ select version, tree_level,
from pgstatindex('test_pkey');
version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation
---------+------------+------------+---------------+----------------+------------+-------------+---------------+------------------+--------------------
- 3 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
+ 4 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
(1 row)
select version, tree_level,
@@ -58,7 +58,7 @@ select version, tree_level,
from pgstatindex('test_pkey'::text);
version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation
---------+------------+------------+---------------+----------------+------------+-------------+---------------+------------------+--------------------
- 3 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
+ 4 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
(1 row)
select version, tree_level,
@@ -68,7 +68,7 @@ select version, tree_level,
from pgstatindex('test_pkey'::name);
version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation
---------+------------+------------+---------------+----------------+------------+-------------+---------------+------------------+--------------------
- 3 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
+ 4 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
(1 row)
select version, tree_level,
@@ -78,7 +78,7 @@ select version, tree_level,
from pgstatindex('test_pkey'::regclass);
version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation
---------+------------+------------+---------------+----------------+------------+-------------+---------------+------------------+--------------------
- 3 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
+ 4 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | NaN | NaN
(1 row)
select pg_relpages('test');
@@ -232,7 +232,7 @@ create index test_partition_hash_idx on test_partition using hash (a);
select pgstatindex('test_partition_idx');
pgstatindex
------------------------------
- (3,0,8192,0,0,0,0,0,NaN,NaN)
+ (4,0,8192,0,0,0,0,0,NaN,NaN)
(1 row)
select pgstathashindex('test_partition_hash_idx');
diff --git a/src/backend/access/nbtree/README b/src/backend/access/nbtree/README
index 3680e69b89..dc6c65d201 100644
--- a/src/backend/access/nbtree/README
+++ b/src/backend/access/nbtree/README
@@ -34,30 +34,47 @@ Differences to the Lehman & Yao algorithm
We have made the following changes in order to incorporate the L&Y algorithm
into Postgres:
-The requirement that all btree keys be unique is too onerous,
-but the algorithm won't work correctly without it. Fortunately, it is
-only necessary that keys be unique on a single tree level, because L&Y
-only use the assumption of key uniqueness when re-finding a key in a
-parent page (to determine where to insert the key for a split page).
-Therefore, we can use the link field to disambiguate multiple
-occurrences of the same user key: only one entry in the parent level
-will be pointing at the page we had split. (Indeed we need not look at
-the real "key" at all, just at the link field.) We can distinguish
-items at the leaf level in the same way, by examining their links to
-heap tuples; we'd never have two items for the same heap tuple.
-
-Lehman and Yao assume that the key range for a subtree S is described
-by Ki < v <= Ki+1 where Ki and Ki+1 are the adjacent keys in the parent
-page. This does not work for nonunique keys (for example, if we have
-enough equal keys to spread across several leaf pages, there *must* be
-some equal bounding keys in the first level up). Therefore we assume
-Ki <= v <= Ki+1 instead. A search that finds exact equality to a
-bounding key in an upper tree level must descend to the left of that
-key to ensure it finds any equal keys in the preceding page. An
-insertion that sees the high key of its target page is equal to the key
-to be inserted has a choice whether or not to move right, since the new
-key could go on either page. (Currently, we try to find a page where
-there is room for the new key without a split.)
+The requirement that all btree keys be unique is satisfied by treating
+heap TID as a tie-breaker attribute. Logical duplicates are sorted in
+descending item pointer order. We don't use btree keys to
+disambiguate downlinks from the internal pages during a page split,
+though: only one entry in the parent level will be pointing at the
+page we just split, so the link fields can be used to re-find
+downlinks in the parent via a linear search.
+
+Lehman and Yao require that the key range for a subtree S is described
+by Ki < v <= Ki+1 where Ki and Ki+1 are the adjacent keys in the
+parent page, but do not account for the need to search the tree based
+only on leading index attributes in a composite index. Since heap TID
+is always used to make btree keys unique (even in unique indexes),
+every btree index is treated as a composite index internally. A
+search that finds exact equality to a pivot tuple in an upper tree
+level must descend to the left of that key to ensure it finds any
+equal keys, even when scan values were provided for all attributes.
+An insertion that sees that the high key of its target page is equal
+to the key to be inserted cannot move right, since the downlink for
+the right sibling in the parent must always be strictly less than
+right sibling keys (this is always possible because the leftmost
+downlink on any non-leaf level is always a negative infinity
+downlink).
+
+We might be able to avoid moving left in the event of a full match on
+all attributes up to and including the heap TID attribute, but that
+would be a very narrow win, since it's rather unlikely that heap TID
+will be an exact match. We can avoid moving left unnecessarily when
+all user-visible keys are equal by avoiding exact equality; a
+sentinel value that's less than any possible heap TID is used by most
+index scans. This is effective because of suffix truncation. An
+"extra" heap TID attribute in pivot tuples is almost always avoided.
+All truncated attributes compare as minus infinity, even against a
+sentinel value, and the sentinel value is less than any real TID
+value, so an unnecessary move to the left is avoided regardless of
+whether or not a heap TID is present in the otherwise-equal pivot
+tuple. Consistently moving left on full equality is also needed by
+page deletion, which re-finds a leaf page by descending the tree while
+searching on the leaf page's high key. If we wanted to avoid moving
+left without breaking page deletion, we'd have to avoid suffix
+truncation, which could never be worth it.
Lehman and Yao don't require read locks, but assume that in-memory
copies of tree pages are unshared. Postgres shares in-memory buffers
@@ -610,21 +627,25 @@ scanned to decide whether to return the entry and whether the scan can
stop (see _bt_checkkeys()).
We use term "pivot" index tuples to distinguish tuples which don't point
-to heap tuples, but rather used for tree navigation. Pivot tuples includes
-all tuples on non-leaf pages and high keys on leaf pages. Note that pivot
-index tuples are only used to represent which part of the key space belongs
-on each page, and can have attribute values copied from non-pivot tuples
-that were deleted and killed by VACUUM some time ago. In principle, we could
-truncate away attributes that are not needed for a page high key during a leaf
-page split, provided that the remaining attributes distinguish the last index
-tuple on the post-split left page as belonging on the left page, and the first
-index tuple on the post-split right page as belonging on the right page. This
-optimization is sometimes called suffix truncation, and may appear in a future
-release. Since the high key is subsequently reused as the downlink in the
-parent page for the new right page, suffix truncation can increase index
-fan-out considerably by keeping pivot tuples short. INCLUDE indexes similarly
-truncate away non-key attributes at the time of a leaf page split,
-increasing fan-out.
+to heap tuples, that are used only for tree navigation. Pivot tuples
+includes all tuples on non-leaf pages and high keys on leaf pages. Note
+that pivot index tuples are only used to represent which part of the key
+space belongs on each page, and can have attribute values copied from
+non-pivot tuples that were deleted and killed by VACUUM some time ago.
+
+We truncate away attributes that are not needed for a page high key during
+a leaf page split, provided that the remaining attributes distinguish the
+last index tuple on the post-split left page as belonging on the left
+page, and the first index tuple on the post-split right page as belonging
+on the right page. A truncated tuple logically retains the truncated
+suffix key attributes, which implicitly have "negative infinity" as their
+value. This optimization is called suffix truncation. Since the high key
+is subsequently reused as the downlink in the parent page for the new
+right page, suffix truncation can increase index fan-out considerably by
+keeping pivot tuples short. INCLUDE indexes are guaranteed to have
+non-key attributes truncated at the time of a leaf page split, but may
+also have some key attributes truncated away, based on the usual criteria
+for key attributes.
Notes About Data Representation
-------------------------------
@@ -658,4 +679,19 @@ downlink. The first data item on each such page has no lower bound
routines must treat it accordingly. The actual key stored in the
item is irrelevant, and need not be stored at all. This arrangement
corresponds to the fact that an L&Y non-leaf page has one more pointer
-than key.
+than key. Suffix truncation's negative infinity attributes behave in
+the same way.
+
+Non-leaf pages only truly need to truncate their first item to zero
+attributes at the leftmost level, since that truly is negative infinity.
+All other negative infinity items are only really negative infinity
+within the subtree that the page is at the root of (or is a leftmost
+page within). We truncate away all attributes of the first item on
+non-leaf pages just the same, to save a little space. If we ever
+avoided zero-truncating items on pages where that doesn't accurately
+represent the absolute separation of the keyspace, we'd be left with
+"low key" items on internal pages -- a key value that can be used as a
+lower bound on items on the page, much like the high key is an upper
+bound. (Actually, that would even be true of "true" negative infinity
+items. One can think of rightmost pages as implicitly containing
+"positive infinity" high keys.)
diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c
index 4b2b4746f7..7d0556b91d 100644
--- a/src/backend/access/nbtree/nbtinsert.c
+++ b/src/backend/access/nbtree/nbtinsert.c
@@ -76,9 +76,11 @@ static Buffer _bt_split(Relation rel, Buffer buf, Buffer cbuf,
static void _bt_insert_parent(Relation rel, Buffer buf, Buffer rbuf,
BTStack stack, bool is_root, bool is_only);
static OffsetNumber _bt_findsplitloc(Relation rel, Page page,
+ bool disjunctpass,
OffsetNumber newitemoff,
Size newitemsz,
bool *newitemonleft);
+static bool _bt_split_isdisjunct(Page page, OffsetNumber item);
static void _bt_checksplitloc(FindSplitData *state,
OffsetNumber firstoldonright, bool newitemonleft,
int dataitemstoleft, Size firstoldonrightsz);
@@ -113,9 +115,12 @@ _bt_doinsert(Relation rel, IndexTuple itup,
bool is_unique = false;
int indnkeyatts;
ScanKey itup_scankey;
+ ItemPointer itup_scantid;
BTStack stack = NULL;
Buffer buf;
OffsetNumber offset;
+ Page page;
+ BTPageOpaque lpageop;
bool fastpath;
indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
@@ -123,6 +128,8 @@ _bt_doinsert(Relation rel, IndexTuple itup,
/* we need an insertion scan key to do our search, so build one */
itup_scankey = _bt_mkscankey(rel, itup);
+ /* we use a heap TID with scan key if this isn't unique case */
+ itup_scantid = (checkUnique == UNIQUE_CHECK_NO ? &itup->t_tid : NULL);
/*
* It's very common to have an index on an auto-incremented or
@@ -149,8 +156,6 @@ top:
if (RelationGetTargetBlock(rel) != InvalidBlockNumber)
{
Size itemsz;
- Page page;
- BTPageOpaque lpageop;
/*
* Conditionally acquire exclusive lock on the buffer before doing any
@@ -180,7 +185,7 @@ top:
!P_IGNORE(lpageop) &&
(PageGetFreeSpace(page) > itemsz) &&
PageGetMaxOffsetNumber(page) >= P_FIRSTDATAKEY(lpageop) &&
- _bt_compare(rel, indnkeyatts, itup_scankey, page,
+ _bt_compare(rel, indnkeyatts, itup_scankey, itup_scantid, page,
P_FIRSTDATAKEY(lpageop)) > 0)
{
/*
@@ -217,8 +222,8 @@ top:
if (!fastpath)
{
/* find the first page containing this key */
- stack = _bt_search(rel, indnkeyatts, itup_scankey, false, &buf, BT_WRITE,
- NULL);
+ stack = _bt_search(rel, indnkeyatts, itup_scankey, itup_scantid, false,
+ &buf, BT_WRITE, NULL);
/* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
@@ -231,8 +236,8 @@ top:
* need to move right in the tree. See Lehman and Yao for an
* excruciatingly precise description.
*/
- buf = _bt_moveright(rel, buf, indnkeyatts, itup_scankey, false,
- true, stack, BT_WRITE, NULL);
+ buf = _bt_moveright(rel, buf, indnkeyatts, itup_scankey, itup_scantid,
+ false, true, stack, BT_WRITE, NULL);
}
/*
@@ -242,12 +247,13 @@ top:
* NOTE: obviously, _bt_check_unique can only detect keys that are already
* in the index; so it cannot defend against concurrent insertions of the
* same key. We protect against that by means of holding a write lock on
- * the target page. Any other would-be inserter of the same key must
- * acquire a write lock on the same target page, so only one would-be
- * inserter can be making the check at one time. Furthermore, once we are
- * past the check we hold write locks continuously until we have performed
- * our insertion, so no later inserter can fail to see our insertion.
- * (This requires some care in _bt_findinsertloc.)
+ * the first page the value could be on, regardless of the value of its
+ * implicit heap TID tie-breaker attribute. Any other would-be inserter of
+ * the same key must acquire a write lock on the same page, so only one
+ * would-be inserter can be making the check at one time. Furthermore,
+ * once we are past the check we hold write locks continuously until we
+ * have performed our insertion, so no later inserter can fail to see our
+ * insertion. (This requires some care in _bt_findinsertloc.)
*
* If we must wait for another xact, we release the lock while waiting,
* and then must start over completely.
@@ -261,7 +267,11 @@ top:
TransactionId xwait;
uint32 speculativeToken;
- offset = _bt_binsrch(rel, buf, indnkeyatts, itup_scankey, false);
+ page = BufferGetPage(buf);
+ lpageop = (BTPageOpaque) PageGetSpecialPointer(page);
+ Assert(itup_scantid == NULL);
+ offset = _bt_binsrch(rel, buf, indnkeyatts, itup_scankey, NULL,
+ P_FIRSTDATAKEY(lpageop), false);
xwait = _bt_check_unique(rel, itup, heapRel, buf, offset, itup_scankey,
checkUnique, &is_unique, &speculativeToken);
@@ -299,7 +309,7 @@ top:
* attributes are not considered part of the key space.
*/
CheckForSerializableConflictIn(rel, NULL, buf);
- /* do the insertion */
+ /* do the insertion, possibly on a page to the right in unique case */
_bt_findinsertloc(rel, &buf, &offset, indnkeyatts, itup_scankey, itup,
stack, heapRel);
_bt_insertonpg(rel, buf, InvalidBuffer, stack, itup, offset, false);
@@ -564,11 +574,11 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
offset = OffsetNumberNext(offset);
else
{
- /* If scankey == hikey we gotta check the next page too */
+ /* If scankey <= hikey we gotta check the next page too */
if (P_RIGHTMOST(opaque))
break;
- if (!_bt_isequal(itupdesc, page, P_HIKEY,
- indnkeyatts, itup_scankey))
+ /* _bt_isequal()'s special NULL semantics not required here */
+ if (_bt_compare(rel, indnkeyatts, itup_scankey, NULL, page, P_HIKEY) > 0)
break;
/* Advance to next non-dead page --- there must be one */
for (;;)
@@ -612,31 +622,22 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
/*
* _bt_findinsertloc() -- Finds an insert location for a tuple
*
- * If the new key is equal to one or more existing keys, we can
- * legitimately place it anywhere in the series of equal keys --- in fact,
- * if the new key is equal to the page's "high key" we can place it on
- * the next page. If it is equal to the high key, and there's not room
- * to insert the new tuple on the current page without splitting, then
- * we can move right hoping to find more free space and avoid a split.
- * (We should not move right indefinitely, however, since that leads to
- * O(N^2) insertion behavior in the presence of many equal keys.)
- * Once we have chosen the page to put the key on, we'll insert it before
- * any existing equal keys because of the way _bt_binsrch() works.
- *
- * If there's not enough room in the space, we try to make room by
- * removing any LP_DEAD tuples.
- *
* On entry, *bufptr and *offsetptr point to the first legal position
- * where the new tuple could be inserted. The caller should hold an
- * exclusive lock on *bufptr. *offsetptr can also be set to
- * InvalidOffsetNumber, in which case the function will search for the
- * right location within the page if needed. On exit, they point to the
- * chosen insert location. If _bt_findinsertloc decides to move right,
- * the lock and pin on the original page will be released and the new
- * page returned to the caller is exclusively locked instead.
+ * where the new tuple could be inserted if we were to treat it as having
+ * no implicit heap TID; only callers that just called _bt_check_unique()
+ * provide this hint (all other callers should set *offsetptr to
+ * InvalidOffsetNumber). The caller should hold an exclusive lock on
+ * *bufptr in all cases. On exit, they both point to the chosen insert
+ * location in all cases. If _bt_findinsertloc decides to move right, the
+ * lock and pin on the original page will be released, and the new page
+ * returned to the caller is exclusively locked instead.
+ *
+ * This is also where opportunistic microvacuuming of LP_DEAD tuples
+ * occurs.
*
* newtup is the new tuple we're inserting, and scankey is an insertion
- * type scan key for it.
+ * type scan key for it. We take a "scantid" heap TID attribute value
+ * from newtup directly.
*/
static void
_bt_findinsertloc(Relation rel,
@@ -652,9 +653,9 @@ _bt_findinsertloc(Relation rel,
Page page = BufferGetPage(buf);
Size itemsz;
BTPageOpaque lpageop;
- bool movedright,
- vacuumed;
+ bool hintinvalidated;
OffsetNumber newitemoff;
+ OffsetNumber lowitemoff;
OffsetNumber firstlegaloff = *offsetptr;
lpageop = (BTPageOpaque) PageGetSpecialPointer(page);
@@ -684,59 +685,30 @@ _bt_findinsertloc(Relation rel,
errtableconstraint(heapRel,
RelationGetRelationName(rel))));
- /*----------
- * If we will need to split the page to put the item on this page,
- * check whether we can put the tuple somewhere to the right,
- * instead. Keep scanning right until we
- * (a) find a page with enough free space,
- * (b) reach the last page where the tuple can legally go, or
- * (c) get tired of searching.
- * (c) is not flippant; it is important because if there are many
- * pages' worth of equal keys, it's better to split one of the early
- * pages than to scan all the way to the end of the run of equal keys
- * on every insert. We implement "get tired" as a random choice,
- * since stopping after scanning a fixed number of pages wouldn't work
- * well (we'd never reach the right-hand side of previously split
- * pages). Currently the probability of moving right is set at 0.99,
- * which may seem too high to change the behavior much, but it does an
- * excellent job of preventing O(N^2) behavior with many equal keys.
- *----------
+ /* firstlegaloff/offsetptr hint (if any) assumed valid initially */
+ hintinvalidated = false;
+
+ /*
+ * TODO: Restore the logic for finding a page to insert on in the event of
+ * many duplicates for pre-pg_upgrade indexes. The whole search through
+ * pages of logical duplicates to determine where to insert seems like
+ * something that has little upside, but that doesn't make it okay to
+ * ignore the performance characteristics after pg_upgrade is run, but
+ * before a REINDEX can run to bump BTREE_VERSION.
*/
- movedright = false;
- vacuumed = false;
- while (PageGetFreeSpace(page) < itemsz)
+ while (true)
{
Buffer rbuf;
BlockNumber rblkno;
- /*
- * before considering moving right, see if we can obtain enough space
- * by erasing LP_DEAD items
- */
- if (P_ISLEAF(lpageop) && P_HAS_GARBAGE(lpageop))
- {
- _bt_vacuum_one_page(rel, buf, heapRel);
-
- /*
- * remember that we vacuumed this page, because that makes the
- * hint supplied by the caller invalid
- */
- vacuumed = true;
-
- if (PageGetFreeSpace(page) >= itemsz)
- break; /* OK, now we have enough space */
- }
-
- /*
- * nope, so check conditions (b) and (c) enumerated above
- */
if (P_RIGHTMOST(lpageop) ||
- _bt_compare(rel, keysz, scankey, page, P_HIKEY) != 0 ||
- random() <= (MAX_RANDOM_VALUE / 100))
+ _bt_compare(rel, keysz, scankey, &newtup->t_tid, page, P_HIKEY) <= 0)
break;
/*
- * step right to next non-dead page
+ * step right to next non-dead page. this is only needed for unique
+ * indexes, and pg_upgrade'd indexes that still use BTREE_VERSION 2 or
+ * 3, where heap TID isn't considered to be a part of the keyspace.
*
* must write-lock that page before releasing write lock on current
* page; else someone else's _bt_check_unique scan could fail to see
@@ -775,24 +747,40 @@ _bt_findinsertloc(Relation rel,
}
_bt_relbuf(rel, buf);
buf = rbuf;
- movedright = true;
- vacuumed = false;
+ hintinvalidated = true;
}
+ Assert(P_ISLEAF(lpageop));
+
/*
- * Now we are on the right page, so find the insert position. If we moved
- * right at all, we know we should insert at the start of the page. If we
- * didn't move right, we can use the firstlegaloff hint if the caller
- * supplied one, unless we vacuumed the page which might have moved tuples
- * around making the hint invalid. If we didn't move right or can't use
- * the hint, find the position by searching.
+ * Perform micro-vacuuming of the page we're about to insert tuple on to if
+ * it looks like it has LP_DEAD items.
*/
- if (movedright)
- newitemoff = P_FIRSTDATAKEY(lpageop);
- else if (firstlegaloff != InvalidOffsetNumber && !vacuumed)
- newitemoff = firstlegaloff;
+ if (P_HAS_GARBAGE(lpageop) && PageGetFreeSpace(page) < itemsz)
+ {
+ _bt_vacuum_one_page(rel, buf, heapRel);
+
+ hintinvalidated = true;
+ }
+
+ /*
+ * Consider using caller's hint to avoid repeated binary search effort.
+ *
+ * Note that the hint is only provided by callers that checked uniqueness.
+ * The hint is used as a lower bound for a new binary search, since
+ * caller's original binary search won't have specified a scan tid.
+ */
+ if (firstlegaloff == InvalidOffsetNumber || hintinvalidated)
+ lowitemoff = P_FIRSTDATAKEY(lpageop);
else
- newitemoff = _bt_binsrch(rel, buf, keysz, scankey, false);
+ {
+ Assert(firstlegaloff == _bt_binsrch(rel, buf, keysz, scankey, NULL,
+ P_FIRSTDATAKEY(lpageop), false));
+ lowitemoff = firstlegaloff;
+ }
+
+ newitemoff = _bt_binsrch(rel, buf, keysz, scankey, &newtup->t_tid,
+ lowitemoff, false);
*bufptr = buf;
*offsetptr = newitemoff;
@@ -851,11 +839,12 @@ _bt_insertonpg(Relation rel,
/* child buffer must be given iff inserting on an internal page */
Assert(P_ISLEAF(lpageop) == !BufferIsValid(cbuf));
/* tuple must have appropriate number of attributes */
+ Assert(BTreeTupleGetNAtts(itup, rel) > 0);
Assert(!P_ISLEAF(lpageop) ||
BTreeTupleGetNAtts(itup, rel) ==
IndexRelationGetNumberOfAttributes(rel));
Assert(P_ISLEAF(lpageop) ||
- BTreeTupleGetNAtts(itup, rel) ==
+ BTreeTupleGetNAtts(itup, rel) <=
IndexRelationGetNumberOfKeyAttributes(rel));
/* The caller should've finished any incomplete splits already. */
@@ -900,7 +889,7 @@ _bt_insertonpg(Relation rel,
BlockNumberIsValid(RelationGetTargetBlock(rel))));
/* Choose the split point */
- firstright = _bt_findsplitloc(rel, page,
+ firstright = _bt_findsplitloc(rel, page, true,
newitemoff, itemsz,
&newitemonleft);
@@ -1143,8 +1132,6 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
OffsetNumber i;
bool isleaf;
IndexTuple lefthikey;
- int indnatts = IndexRelationGetNumberOfAttributes(rel);
- int indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
/* Acquire a new page to split into */
rbuf = _bt_getbuf(rel, P_NEW, BT_WRITE);
@@ -1214,7 +1201,9 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
itemid = PageGetItemId(origpage, P_HIKEY);
itemsz = ItemIdGetLength(itemid);
item = (IndexTuple) PageGetItem(origpage, itemid);
- Assert(BTreeTupleGetNAtts(item, rel) == indnkeyatts);
+ Assert(BTreeTupleGetNAtts(item, rel) > 0);
+ Assert(BTreeTupleGetNAtts(item, rel) <=
+ IndexRelationGetNumberOfKeyAttributes(rel));
if (PageAddItem(rightpage, (Item) item, itemsz, rightoff,
false, false) == InvalidOffsetNumber)
{
@@ -1228,8 +1217,9 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
/*
* The "high key" for the new left page will be the first key that's going
- * to go into the new right page. This might be either the existing data
- * item at position firstright, or the incoming tuple.
+ * to go into the new right page, or possibly a truncated version if this
+ * is a leaf page split. This might be either the existing data item at
+ * position firstright, or the incoming tuple.
*/
leftoff = P_HIKEY;
if (!newitemonleft && newitemoff == firstright)
@@ -1247,25 +1237,77 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
}
/*
- * Truncate non-key (INCLUDE) attributes of the high key item before
- * inserting it on the left page. This only needs to happen at the leaf
- * level, since in general all pivot tuple values originate from leaf
- * level high keys. This isn't just about avoiding unnecessary work,
- * though; truncating unneeded key attributes (more aggressive suffix
- * truncation) can only be performed at the leaf level anyway. This is
- * because a pivot tuple in a grandparent page must guide a search not
- * only to the correct parent page, but also to the correct leaf page.
+ * Truncate attributes of the high key item before inserting it on the left
+ * page. This can only happen at the leaf level, since in general all
+ * pivot tuple values originate from leaf level high keys. This isn't just
+ * about avoiding unnecessary work, though; truncating unneeded key suffix
+ * attributes can only be performed at the leaf level anyway. This is
+ * because a pivot tuple in a grandparent page must guide a search not only
+ * to the correct parent page, but also to the correct leaf page.
+ *
+ * Note that non-key (INCLUDE) attributes are always truncated away here.
+ * Additional key attributes are truncated away when they're not required
+ * to correctly separate the key space.
*/
- if (indnatts != indnkeyatts && isleaf)
+ if (isleaf)
{
- lefthikey = _bt_nonkey_truncate(rel, item);
+ OffsetNumber lastleftoff;
+ IndexTuple lastleft;
+
+ /*
+ * Determine which tuple is on the left side of the split point, and
+ * generate truncated copy of the right tuple. Truncate as
+ * aggressively as possible without generating a high key for the left
+ * side of the split (and later downlink for the right side) that fails
+ * to distinguish each side. The new high key needs to be strictly
+ * less than all tuples on the right side of the split, but can be
+ * equal to items on the left side of the split. We almost always find
+ * a way to make it strictly greater than lastleft, though.
+ *
+ * Handle the case where the incoming tuple is about to become the last
+ * item on the left side of the split.
+ */
+ if (newitemonleft && newitemoff == firstright)
+ lastleft = newitem;
+ else
+ {
+ lastleftoff = OffsetNumberPrev(firstright);
+ itemid = PageGetItemId(origpage, lastleftoff);
+ lastleft = (IndexTuple) PageGetItem(origpage, itemid);
+ }
+
+ Assert(lastleft != item);
+ lefthikey = _bt_suffix_truncate(rel, lastleft, item);
itemsz = IndexTupleSize(lefthikey);
itemsz = MAXALIGN(itemsz);
+#ifdef DEBUG_SPLITS
+ {
+ TupleDesc itupdesc = RelationGetDescr(rel);
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ char *lastleftstr;
+ char *firstrightstr;
+
+ index_deform_tuple(lastleft, itupdesc, values, isnull);
+ lastleftstr = BuildIndexValueDescription(rel, values, isnull);
+ index_deform_tuple(item, itupdesc, values, isnull);
+ firstrightstr = BuildIndexValueDescription(rel, values, isnull);
+
+ elog(LOG, "split of block %u "
+ "last left %s first right %s "
+ "attributes in new high key for left page %u%s",
+ BufferGetBlockNumber(buf), lastleftstr, firstrightstr,
+ BTreeTupleGetNAtts(lefthikey, rel),
+ BTreeTupleGetHeapTID(lefthikey) != NULL ? " (plus heap TID)":"");
+ }
+#endif
}
else
lefthikey = item;
- Assert(BTreeTupleGetNAtts(lefthikey, rel) == indnkeyatts);
+ Assert(BTreeTupleGetNAtts(lefthikey, rel) > 0);
+ Assert(BTreeTupleGetNAtts(lefthikey, rel) <=
+ IndexRelationGetNumberOfKeyAttributes(rel));
if (PageAddItem(leftpage, (Item) lefthikey, itemsz, leftoff,
false, false) == InvalidOffsetNumber)
{
@@ -1458,7 +1500,6 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
xl_btree_split xlrec;
uint8 xlinfo;
XLogRecPtr recptr;
- bool loglhikey = false;
xlrec.level = ropaque->btpo.level;
xlrec.firstright = firstright;
@@ -1487,22 +1528,10 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
if (newitemonleft)
XLogRegisterBufData(0, (char *) newitem, MAXALIGN(newitemsz));
- /* Log left page */
- if (!isleaf || indnatts != indnkeyatts)
- {
- /*
- * We must also log the left page's high key. There are two
- * reasons for that: right page's leftmost key is suppressed on
- * non-leaf levels and in covering indexes included columns are
- * truncated from high keys. Show it as belonging to the left
- * page buffer, so that it is not stored if XLogInsert decides it
- * needs a full-page image of the left page.
- */
- itemid = PageGetItemId(origpage, P_HIKEY);
- item = (IndexTuple) PageGetItem(origpage, itemid);
- XLogRegisterBufData(0, (char *) item, MAXALIGN(IndexTupleSize(item)));
- loglhikey = true;
- }
+ /* Log left page. We must also log the left page's high key. */
+ itemid = PageGetItemId(origpage, P_HIKEY);
+ item = (IndexTuple) PageGetItem(origpage, itemid);
+ XLogRegisterBufData(0, (char *) item, MAXALIGN(IndexTupleSize(item)));
/*
* Log the contents of the right page in the format understood by
@@ -1520,9 +1549,7 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
(char *) rightpage + ((PageHeader) rightpage)->pd_upper,
((PageHeader) rightpage)->pd_special - ((PageHeader) rightpage)->pd_upper);
- xlinfo = newitemonleft ?
- (loglhikey ? XLOG_BTREE_SPLIT_L_HIGHKEY : XLOG_BTREE_SPLIT_L) :
- (loglhikey ? XLOG_BTREE_SPLIT_R_HIGHKEY : XLOG_BTREE_SPLIT_R);
+ xlinfo = newitemonleft ? XLOG_BTREE_SPLIT_L : XLOG_BTREE_SPLIT_R;
recptr = XLogInsert(RM_BTREE_ID, xlinfo);
PageSetLSN(origpage, recptr);
@@ -1575,10 +1602,31 @@ _bt_split(Relation rel, Buffer buf, Buffer cbuf, OffsetNumber firstright,
* righthand page, plus a boolean indicating whether the new tuple goes on
* the left or right page. The bool is necessary to disambiguate the case
* where firstright == newitemoff.
+ *
+ * The top-level caller should specify disjunctpass as true. This makes
+ * finding a split location refuse to consider candidate split points that
+ * will result in a non-discriminating downlink for the right half of the
+ * split in the parent page. If we fail to find a split point this way, we
+ * call ourselves recursively without insisting on a disjunct split point.
+ * This avoids cases where suffix truncation must add an extra heap TID to
+ * the new pivot during a leaf page split. It also makes it far less likely
+ * that any pivot tuples that have a heap attribute TID added despite our
+ * best efforts will get used as a downlink a second time, in the
+ * grandparent. There should only be downlinks with an extra heap TID
+ * attribute in pages at higher levels of the tree when a very large
+ * proportion of leaf tuples happen to be logical duplicates.
+ *
+ * The disjunctpass strategy may seem very aggressive, as it could lead to
+ * each half of the split having very different amounts of free space.
+ * Note, however, that logical duplicate downlinks in internal pages can
+ * only help _bt_search() callers that pass a scantid argument (i.e.
+ * UNIQUE_CHECK_NO inserters). Even there, the effect on performance ought
+ * to be self-limiting.
*/
static OffsetNumber
_bt_findsplitloc(Relation rel,
Page page,
+ bool disjunctpass,
OffsetNumber newitemoff,
Size newitemsz,
bool *newitemonleft)
@@ -1661,6 +1709,14 @@ _bt_findsplitloc(Relation rel,
itemid = PageGetItemId(page, offnum);
itemsz = MAXALIGN(ItemIdGetLength(itemid)) + sizeof(ItemIdData);
+ /*
+ * FIXME: This doesn't consider the case where the tuple to be
+ * inserted is placed immediately before or after item.
+ */
+ if (disjunctpass && offnum > P_FIRSTDATAKEY(opaque) &&
+ !_bt_split_isdisjunct(page, offnum))
+ goto skipnondisjunct;
+
/*
* Will the new item go to left or right of split?
*/
@@ -1688,6 +1744,8 @@ _bt_findsplitloc(Relation rel,
break;
}
+skipnondisjunct:
+
olddataitemstoleft += itemsz;
}
@@ -1699,18 +1757,80 @@ _bt_findsplitloc(Relation rel,
if (newitemoff > maxoff && !goodenoughfound)
_bt_checksplitloc(&state, newitemoff, false, olddataitemstotal, 0);
- /*
- * I believe it is not possible to fail to find a feasible split, but just
- * in case ...
- */
if (!state.have_split)
+ {
+ /*
+ * We couldn't find a split point using disjunctpass strategy. Take
+ * a second pass at it, this time only weighing space utilization on
+ * the page that we're about to split.
+ */
+ if (disjunctpass)
+ return _bt_findsplitloc(rel, page, false, newitemoff, newitemsz,
+ newitemonleft);
+ /*
+ * I believe it is not possible to fail to find a feasible split, but
+ * just in case ...
+ */
elog(ERROR, "could not find a feasible split point for index \"%s\"",
RelationGetRelationName(rel));
+ }
*newitemonleft = state.newitemonleft;
return state.firstright;
}
+/*
+ * _bt_split_isdisjunct - does split point separate key space well?
+ *
+ * firstoldonright is a candidate split point.
+ *
+ * Checks if point between the would-be last left and first right tuples on the
+ * hypothetical post-split halves are disjunct.
+ */
+static bool
+_bt_split_isdisjunct(Page page, OffsetNumber firstoldonright)
+{
+ BTPageOpaque opaque;
+ ItemId itemid;
+ IndexTuple itup;
+
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ itemid = PageGetItemId(page, firstoldonright);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+
+ if (P_ISLEAF(opaque))
+ {
+ IndexTuple lastitup;
+ char *curTup;
+ char *lastTup;
+ Size datasz;
+
+ /*
+ * Use memcmp() to cheaply approximate real comparisons.
+ *
+ * XXX: This clearly needs more work, especially to make things like
+ * TPC-C's order_lines pkey make better use of suffix truncation.
+ *
+ * FIXME: What about covering index pivot tuples? Clearly using
+ * memcmp() there does not approximate comparing key attributes very
+ * well at all.
+ */
+ itemid = PageGetItemId(page, OffsetNumberPrev(firstoldonright));
+ lastitup = (IndexTuple) PageGetItem(page, itemid);
+ lastTup = (char *) lastitup + IndexInfoFindDataOffset(lastitup->t_info);
+ curTup = (char *) itup + IndexInfoFindDataOffset(itup->t_info);
+ datasz = IndexTupleSize(itup) - IndexInfoFindDataOffset(itup->t_info);
+
+ if (IndexTupleSize(lastitup) == IndexTupleSize(itup) &&
+ memcmp(lastTup, curTup, datasz) == 0)
+ return false;
+ }
+ else if (BTreeTupleGetHeapTID(itup) != NULL)
+ return false;
+
+ return true;
+}
+
/*
* Subroutine to analyze a particular possible split choice (ie, firstright
* and newitemonleft settings), and record the best split so far in *state.
@@ -1761,6 +1881,9 @@ _bt_checksplitloc(FindSplitData *state,
*/
leftfree -= firstrightitemsz;
+ /* Charge for case where a heap TID is added */
+ leftfree -= MAXALIGN(sizeof(ItemPointerData));
+
/* account for the new item */
if (newitemonleft)
leftfree -= (int) state->newitemsz;
@@ -2210,7 +2333,8 @@ _bt_newroot(Relation rel, Buffer lbuf, Buffer rbuf)
/*
* insert the right page pointer into the new root page.
*/
- Assert(BTreeTupleGetNAtts(right_item, rel) ==
+ Assert(BTreeTupleGetNAtts(right_item, rel) > 0);
+ Assert(BTreeTupleGetNAtts(right_item, rel) <=
IndexRelationGetNumberOfKeyAttributes(rel));
if (PageAddItem(rootpage, (Item) right_item, right_item_sz, P_FIRSTKEY,
false, false) == InvalidOffsetNumber)
@@ -2322,8 +2446,8 @@ _bt_pgaddtup(Page page,
/*
* _bt_isequal - used in _bt_doinsert in check for duplicates.
*
- * This is very similar to _bt_compare, except for NULL handling.
- * Rule is simple: NOT_NULL not equal NULL, NULL not equal NULL too.
+ * This is very similar to _bt_compare, except for NULL and negative infinity
+ * handling. Rule is simple: NOT_NULL not equal NULL, NULL not equal NULL too.
*/
static bool
_bt_isequal(TupleDesc itupdesc, Page page, OffsetNumber offnum,
@@ -2337,12 +2461,6 @@ _bt_isequal(TupleDesc itupdesc, Page page, OffsetNumber offnum,
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
- /*
- * It's okay that we might perform a comparison against a truncated page
- * high key when caller needs to determine if _bt_check_unique scan must
- * continue on to the next page. Caller never asks us to compare non-key
- * attributes within an INCLUDE index.
- */
for (i = 1; i <= keysz; i++)
{
AttrNumber attno;
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 4082103fe2..f63615341c 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -1421,10 +1421,12 @@ _bt_pagedel(Relation rel, Buffer buf)
/* we need an insertion scan key for the search, so build one */
itup_scankey = _bt_mkscankey(rel, targetkey);
- /* find the leftmost leaf page containing this key */
+ /* get stack to leaf page by searching index */
stack = _bt_search(rel,
- IndexRelationGetNumberOfKeyAttributes(rel),
- itup_scankey, false, &lbuf, BT_READ, NULL);
+ BTreeTupleGetNAtts(targetkey, rel),
+ itup_scankey,
+ BTreeTupleGetHeapTID(targetkey), false,
+ &lbuf, BT_READ, NULL);
/* don't need a pin on the page */
_bt_relbuf(rel, lbuf);
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index 0bcfa10b86..b5e22bcb09 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -69,11 +69,13 @@ _bt_drop_lock_and_maybe_pin(IndexScanDesc scan, BTScanPos sp)
/*
- * _bt_search() -- Search the tree for a particular scankey,
+ * _bt_search() -- Search the tree for a particular scankey + scantid,
* or more precisely for the first leaf page it could be on.
*
* The passed scankey must be an insertion-type scankey (see nbtree/README),
- * but it can omit the rightmost column(s) of the index.
+ * but it can omit the rightmost column(s) of the index. The scantid
+ * argument may also be omitted (caller passes NULL), since it's logically
+ * the "real" rightmost attribute.
*
* When nextkey is false (the usual case), we are looking for the first
* item >= scankey. When nextkey is true, we are looking for the first
@@ -94,8 +96,8 @@ _bt_drop_lock_and_maybe_pin(IndexScanDesc scan, BTScanPos sp)
* any incomplete splits encountered during the search will be finished.
*/
BTStack
-_bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
- Buffer *bufP, int access, Snapshot snapshot)
+_bt_search(Relation rel, int keysz, ScanKey scankey, ItemPointer scantid,
+ bool nextkey, Buffer *bufP, int access, Snapshot snapshot)
{
BTStack stack_in = NULL;
@@ -130,7 +132,7 @@ _bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
* if the leaf page is split and we insert to the parent page). But
* this is a good opportunity to finish splits of internal pages too.
*/
- *bufP = _bt_moveright(rel, *bufP, keysz, scankey, nextkey,
+ *bufP = _bt_moveright(rel, *bufP, keysz, scankey, scantid, nextkey,
(access == BT_WRITE), stack_in,
BT_READ, snapshot);
@@ -144,7 +146,8 @@ _bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
* Find the appropriate item on the internal page, and get the child
* page that it points to.
*/
- offnum = _bt_binsrch(rel, *bufP, keysz, scankey, nextkey);
+ offnum = _bt_binsrch(rel, *bufP, keysz, scankey, scantid,
+ P_FIRSTDATAKEY(opaque), nextkey);
itemid = PageGetItemId(page, offnum);
itup = (IndexTuple) PageGetItem(page, itemid);
blkno = BTreeInnerTupleGetDownLink(itup);
@@ -157,8 +160,8 @@ _bt_search(Relation rel, int keysz, ScanKey scankey, bool nextkey,
* downlink (block) to uniquely identify the index entry, in case it
* moves right while we're working lower in the tree. See the paper
* by Lehman and Yao for how this is detected and handled. (We use the
- * child link to disambiguate duplicate keys in the index -- Lehman
- * and Yao disallow duplicate keys.)
+ * child link to disambiguate duplicate keys in the index, which is
+ * faster than comparing the keys themselves.)
*/
new_stack = (BTStack) palloc(sizeof(BTStackData));
new_stack->bts_blkno = par_blkno;
@@ -215,6 +218,7 @@ _bt_moveright(Relation rel,
Buffer buf,
int keysz,
ScanKey scankey,
+ ItemPointer scantid,
bool nextkey,
bool forupdate,
BTStack stack,
@@ -275,7 +279,7 @@ _bt_moveright(Relation rel,
continue;
}
- if (P_IGNORE(opaque) || _bt_compare(rel, keysz, scankey, page, P_HIKEY) >= cmpval)
+ if (P_IGNORE(opaque) || _bt_compare(rel, keysz, scankey, scantid, page, P_HIKEY) >= cmpval)
{
/* step right one page */
buf = _bt_relandgetbuf(rel, buf, opaque->btpo_next, access);
@@ -307,6 +311,12 @@ _bt_moveright(Relation rel,
* particular, this means it is possible to return a value 1 greater than the
* number of keys on the page, if the scankey is > all keys on the page.)
*
+ * Caller passes own low value for binary search. This can be used to
+ * resume a partial binary search without repeated effort. _bt_check_unique
+ * callers use this to avoid repeated work. This only works when a buffer
+ * lock is held throughout, and we're passed a leaf page both times, and
+ * nextkey is false.
+ *
* On an internal (non-leaf) page, _bt_binsrch() returns the OffsetNumber
* of the last key < given scankey, or last key <= given scankey if nextkey
* is true. (Since _bt_compare treats the first data key of such a page as
@@ -324,19 +334,19 @@ _bt_binsrch(Relation rel,
Buffer buf,
int keysz,
ScanKey scankey,
+ ItemPointer scantid,
+ OffsetNumber low,
bool nextkey)
{
Page page;
BTPageOpaque opaque;
- OffsetNumber low,
- high;
+ OffsetNumber high;
int32 result,
cmpval;
page = BufferGetPage(buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
- low = P_FIRSTDATAKEY(opaque);
high = PageGetMaxOffsetNumber(page);
/*
@@ -371,7 +381,7 @@ _bt_binsrch(Relation rel,
/* We have low <= mid < high, so mid points at a real slot */
- result = _bt_compare(rel, keysz, scankey, page, mid);
+ result = _bt_compare(rel, keysz, scankey, scantid, page, mid);
if (result >= cmpval)
low = mid + 1;
@@ -401,20 +411,8 @@ _bt_binsrch(Relation rel,
/*----------
* _bt_compare() -- Compare scankey to a particular tuple on the page.
*
- * The passed scankey must be an insertion-type scankey (see nbtree/README),
- * but it can omit the rightmost column(s) of the index.
- *
- * keysz: number of key conditions to be checked (might be less than the
- * number of index columns!)
- * page/offnum: location of btree item to be compared to.
- *
- * This routine returns:
- * <0 if scankey < tuple at offnum;
- * 0 if scankey == tuple at offnum;
- * >0 if scankey > tuple at offnum.
- * NULLs in the keys are treated as sortable values. Therefore
- * "equality" does not necessarily mean that the item should be
- * returned to the caller as a matching key!
+ * Convenience wrapper for _bt_tuple_compare() callers that want to compare
+ * an offset on a particular page.
*
* CRUCIAL NOTE: on a non-leaf page, the first data key is assumed to be
* "minus infinity": this routine will always claim it is less than the
@@ -428,24 +426,66 @@ int32
_bt_compare(Relation rel,
int keysz,
ScanKey scankey,
+ ItemPointer scantid,
Page page,
OffsetNumber offnum)
{
- TupleDesc itupdesc = RelationGetDescr(rel);
BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
IndexTuple itup;
- int i;
Assert(_bt_check_natts(rel, page, offnum));
/*
* Force result ">" if target item is first data item on an internal page
* --- see NOTE above.
+ *
+ * A minus infinity key has all attributes truncated away, so this test is
+ * redundant with the minus infinity attribute tie-breaker. However, the
+ * number of attributes in minus infinity tuples was not explicitly
+ * represented as 0 until PostgreSQL v11, so an explicit offnum test is
+ * still required.
*/
if (!P_ISLEAF(opaque) && offnum == P_FIRSTDATAKEY(opaque))
return 1;
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
+ return _bt_tuple_compare(rel, keysz, scankey, scantid, itup);
+}
+
+/*----------
+ * _bt_tuple_compare() -- Compare scankey to a particular tuple.
+ *
+ * The passed scankey must be an insertion-type scankey (see nbtree/README),
+ * but it can omit the rightmost column(s) of the index.
+ *
+ * keysz: number of key conditions to be checked (might be less than the
+ * number of index columns!)
+ * page/offnum: location of btree item to be compared to.
+ *
+ * This routine returns:
+ * <0 if scankey < tuple at offnum;
+ * 0 if scankey == tuple at offnum;
+ * >0 if scankey > tuple at offnum.
+ * NULLs in the keys are treated as sortable values. Therefore
+ * "equality" does not necessarily mean that the item should be
+ * returned to the caller as a matching key!
+ *----------
+ */
+int32
+_bt_tuple_compare(Relation rel,
+ int keysz,
+ ScanKey scankey,
+ ItemPointer scantid,
+ IndexTuple itup)
+{
+ TupleDesc itupdesc = RelationGetDescr(rel);
+ ItemPointer heapTid;
+ int ntupatts;
+ int ncmpkey;
+ int i;
+
+ Assert(keysz <= IndexRelationGetNumberOfKeyAttributes(rel));
+ ntupatts = BTreeTupleGetNAtts(itup, rel);
/*
* The scan key is set up with the attribute number associated with each
@@ -459,7 +499,8 @@ _bt_compare(Relation rel,
* _bt_first).
*/
- for (i = 1; i <= keysz; i++)
+ ncmpkey = Min(ntupatts, keysz);
+ for (i = 1; i <= ncmpkey; i++)
{
Datum datum;
bool isNull;
@@ -510,8 +551,31 @@ _bt_compare(Relation rel,
scankey++;
}
- /* if we get here, the keys are equal */
- return 0;
+ /*
+ * Use the number of attributes as a tie-breaker, in order to treat
+ * truncated attributes in index as minus infinity.
+ */
+ if (keysz > ntupatts)
+ return 1;
+
+ /* If caller provided no heap TID tie-breaker for scan, they're equal */
+ if (!scantid)
+ return 0;
+
+ /*
+ * Although it isn't counted as an attribute by BTreeTupleGetNAtts(), heap
+ * TID is an implicit final key attribute that ensures that all index
+ * tuples have a distinct set of key attribute values.
+ *
+ * This is often truncated away in pivot tuples, which makes the attribute
+ * value implicitly negative infinity.
+ */
+ heapTid = BTreeTupleGetHeapTID(itup);
+ if (!heapTid)
+ return 1;
+
+ /* Deliberately invert the order, since TIDs "sort DESC" */
+ return ItemPointerCompare(heapTid, scantid);
}
/*
@@ -540,6 +604,7 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
Relation rel = scan->indexRelation;
BTScanOpaque so = (BTScanOpaque) scan->opaque;
Buffer buf;
+ BTPageOpaque opaque;
BTStack stack;
OffsetNumber offnum;
StrategyNumber strat;
@@ -547,6 +612,8 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
bool goback;
ScanKey startKeys[INDEX_MAX_KEYS];
ScanKeyData scankeys[INDEX_MAX_KEYS];
+ ItemPointer scantid;
+ ItemPointerData minscantid;
ScanKeyData notnullkeys[INDEX_MAX_KEYS];
int keysCount = 0;
int i;
@@ -796,6 +863,7 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
* scankeys[] array, using the keys identified by startKeys[].
*/
Assert(keysCount <= INDEX_MAX_KEYS);
+ scantid = NULL;
for (i = 0; i < keysCount; i++)
{
ScanKey cur = startKeys[i];
@@ -932,6 +1000,30 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
}
}
+ /*
+ * When all key attributes will be in insertion scankey, manufacture
+ * sentinel scan tid that's less than any possible heap TID in the index.
+ * This is still greater than minus infinity to _bt_compare, allowing
+ * _bt_search to follow a downlink with scankey-equal attributes, but a
+ * truncated-away heap TID.
+ *
+ * If we didn't do this then affected index scans would have to
+ * unnecessarily visit an extra page before moving right to the page they
+ * should have landed on from the parent in the first place.
+ *
+ * (Note that implementing this by adding hard-coding to _bt_compare is
+ * unworkable, since some _bt_search callers need to re-find a leaf page
+ * using the page's high key.)
+ */
+ if (keysCount >= IndexRelationGetNumberOfKeyAttributes(rel))
+ {
+ scantid = &minscantid;
+
+ /* Heap TID attribute uses DESC ordering */
+ ItemPointerSetBlockNumber(scantid, InvalidBlockNumber);
+ ItemPointerSetOffsetNumber(scantid, InvalidOffsetNumber);
+ }
+
/*----------
* Examine the selected initial-positioning strategy to determine exactly
* where we need to start the scan, and set flag variables to control the
@@ -1024,11 +1116,11 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
}
/*
- * Use the manufactured insertion scan key to descend the tree and
- * position ourselves on the target leaf page.
+ * Use the manufactured insertion scan key (and possibly a scantid) to
+ * descend the tree and position ourselves on the target leaf page.
*/
- stack = _bt_search(rel, keysCount, scankeys, nextkey, &buf, BT_READ,
- scan->xs_snapshot);
+ stack = _bt_search(rel, keysCount, scankeys, scantid, nextkey, &buf,
+ BT_READ, scan->xs_snapshot);
/* don't need to keep the stack around... */
_bt_freestack(stack);
@@ -1057,7 +1149,9 @@ _bt_first(IndexScanDesc scan, ScanDirection dir)
_bt_initialize_more_data(so, dir);
/* position to the precise item on the page */
- offnum = _bt_binsrch(rel, buf, keysCount, scankeys, nextkey);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(BufferGetPage(buf));
+ offnum = _bt_binsrch(rel, buf, keysCount, scankeys, scantid,
+ P_FIRSTDATAKEY(opaque), nextkey);
/*
* If nextkey = false, we are positioned at the first item >= scan key, or
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index 16f5755777..24bae8a454 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -796,8 +796,6 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
OffsetNumber last_off;
Size pgspc;
Size itupsz;
- int indnatts = IndexRelationGetNumberOfAttributes(wstate->index);
- int indnkeyatts = IndexRelationGetNumberOfKeyAttributes(wstate->index);
/*
* This is a handy place to check for cancel interrupts during the btree
@@ -880,19 +878,28 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
ItemIdSetUnused(ii); /* redundant */
((PageHeader) opage)->pd_lower -= sizeof(ItemIdData);
- if (indnkeyatts != indnatts && P_ISLEAF(opageop))
+ if (P_ISLEAF(opageop))
{
- IndexTuple truncated;
- Size truncsz;
+ IndexTuple lastleft;
+ IndexTuple truncated;
+ Size truncsz;
/*
- * Truncate any non-key attributes from high key on leaf level
- * (i.e. truncate on leaf level if we're building an INCLUDE
- * index). This is only done at the leaf level because downlinks
- * in internal pages are either negative infinity items, or get
- * their contents from copying from one level down. See also:
+ * Truncate away any unneeded attributes from high key on leaf
+ * level. This is only done at the leaf level because downlinks in
+ * internal pages are either negative infinity items, or get their
+ * contents from copying from one level down. See also:
* _bt_split().
*
+ * We don't try to bias our choice of split point to make it more
+ * likely that _bt_suffix_truncate() can truncate away more
+ * attributes, whereas the split point passed to _bt_split() is
+ * chosen much more delicately. Suffix truncation is mostly useful
+ * because it can greatly improve space utilization for workloads
+ * with random insertions. It doesn't seem worthwhile to add
+ * complex logic for choosing a split point here for a benefit that
+ * is bound to be much smaller.
+ *
* Since the truncated tuple is probably smaller than the
* original, it cannot just be copied in place (besides, we want
* to actually save space on the leaf page). We delete the
@@ -905,7 +912,10 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
* the latter portion of the space occupied by the original tuple.
* This is fairly cheap.
*/
- truncated = _bt_nonkey_truncate(wstate->index, oitup);
+ ii = PageGetItemId(opage, OffsetNumberPrev(last_off));
+ lastleft = (IndexTuple) PageGetItem(opage, ii);
+
+ truncated = _bt_suffix_truncate(wstate->index, lastleft, oitup);
truncsz = IndexTupleSize(truncated);
PageIndexTupleDelete(opage, P_HIKEY);
_bt_sortaddtup(opage, truncsz, truncated, P_HIKEY);
@@ -924,8 +934,9 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
if (state->btps_next == NULL)
state->btps_next = _bt_pagestate(wstate, state->btps_level + 1);
- Assert(BTreeTupleGetNAtts(state->btps_minkey, wstate->index) ==
- IndexRelationGetNumberOfKeyAttributes(wstate->index) ||
+ Assert((BTreeTupleGetNAtts(state->btps_minkey, wstate->index) <=
+ IndexRelationGetNumberOfKeyAttributes(wstate->index) &&
+ BTreeTupleGetNAtts(state->btps_minkey, wstate->index) > 0) ||
P_LEFTMOST(opageop));
Assert(BTreeTupleGetNAtts(state->btps_minkey, wstate->index) == 0 ||
!P_LEFTMOST(opageop));
@@ -970,7 +981,7 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup)
* the first item for a page is copied from the prior page in the code
* above. Since the minimum key for an entire level is only used as a
* minus infinity downlink, and never as a high key, there is no need to
- * truncate away non-key attributes at this point.
+ * truncate away suffix attributes at this point.
*/
if (last_off == P_HIKEY)
{
@@ -1029,8 +1040,9 @@ _bt_uppershutdown(BTWriteState *wstate, BTPageState *state)
}
else
{
- Assert(BTreeTupleGetNAtts(s->btps_minkey, wstate->index) ==
- IndexRelationGetNumberOfKeyAttributes(wstate->index) ||
+ Assert((BTreeTupleGetNAtts(s->btps_minkey, wstate->index) <=
+ IndexRelationGetNumberOfKeyAttributes(wstate->index) &&
+ BTreeTupleGetNAtts(s->btps_minkey, wstate->index) > 0) ||
P_LEFTMOST(opaque));
Assert(BTreeTupleGetNAtts(s->btps_minkey, wstate->index) == 0 ||
!P_LEFTMOST(opaque));
@@ -1127,6 +1139,8 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
}
else if (itup != NULL)
{
+ int32 compare = 0;
+
for (i = 1; i <= keysz; i++)
{
SortSupport entry;
@@ -1134,7 +1148,6 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
attrDatum2;
bool isNull1,
isNull2;
- int32 compare;
entry = sortKeys + i - 1;
attrDatum1 = index_getattr(itup, i, tupdes, &isNull1);
@@ -1151,6 +1164,22 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
else if (compare < 0)
break;
}
+
+ /*
+ * If key values are equal, we sort on ItemPointer. This is
+ * required for btree indexes, since heap TID is treated as an
+ * implicit last key attribute in order to ensure that all keys
+ * in the index are physically unique.
+ *
+ * Deliberately invert the order, since TIDs "sort DESC".
+ */
+ if (compare == 0)
+ {
+ compare = ItemPointerCompare(&itup2->t_tid, &itup->t_tid);
+ Assert(compare != 0);
+ if (compare > 0)
+ load1 = false;
+ }
}
else
load1 = false;
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 4528e87c83..4c46decb1d 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -49,6 +49,10 @@ static void _bt_mark_scankey_required(ScanKey skey);
static bool _bt_check_rowcompare(ScanKey skey,
IndexTuple tuple, TupleDesc tupdesc,
ScanDirection dir, bool *continuescan);
+static int _bt_leave_natts(Relation rel, IndexTuple lastleft,
+ IndexTuple firstright);
+static void _bt_set_median_tid(ItemPointer lastleft, ItemPointer firstright,
+ ItemPointer pivotheaptid);
/*
@@ -56,27 +60,34 @@ static bool _bt_check_rowcompare(ScanKey skey,
* Build an insertion scan key that contains comparison data from itup
* as well as comparator routines appropriate to the key datatypes.
*
- * The result is intended for use with _bt_compare().
+ * The result is intended for use with _bt_compare(). If itup has
+ * undergone suffix truncation of key attributes, caller had better
+ * pass BTreeTupleGetNAtts(itup, rel) as keysz to routines like
+ * _bt_search() and _bt_compare() when using returned scan key. This
+ * allows truncated attributes to participate in comparisons (truncated
+ * attributes have implicit negative infinity values). Note that
+ * _bt_compare() never treats a scan key as containing negative
+ * infinity attributes.
*/
ScanKey
_bt_mkscankey(Relation rel, IndexTuple itup)
{
ScanKey skey;
TupleDesc itupdesc;
+ int tupnatts;
int indnatts PG_USED_FOR_ASSERTS_ONLY;
int indnkeyatts;
int16 *indoption;
int i;
itupdesc = RelationGetDescr(rel);
+ tupnatts = BTreeTupleGetNAtts(itup, rel);
indnatts = IndexRelationGetNumberOfAttributes(rel);
indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
indoption = rel->rd_indoption;
- Assert(indnkeyatts > 0);
- Assert(indnkeyatts <= indnatts);
- Assert(BTreeTupleGetNAtts(itup, rel) == indnatts ||
- BTreeTupleGetNAtts(itup, rel) == indnkeyatts);
+ Assert(tupnatts > 0);
+ Assert(tupnatts <= indnatts);
/*
* We'll execute search using scan key constructed on key columns. Non-key
@@ -96,7 +107,21 @@ _bt_mkscankey(Relation rel, IndexTuple itup)
* comparison can be needed.
*/
procinfo = index_getprocinfo(rel, i + 1, BTORDER_PROC);
- arg = index_getattr(itup, i + 1, itupdesc, &null);
+
+ /*
+ * Truncated key attributes may not be represented in index tuple
+ * due to suffix truncation. Keys built from truncated attributes
+ * are defensively represented as NULL values, though they should
+ * still not be allowed to participate in comparisons (caller must
+ * be sure to pass a sane keysz to _bt_compare()).
+ */
+ if (i < tupnatts)
+ arg = index_getattr(itup, i + 1, itupdesc, &null);
+ else
+ {
+ arg = (Datum) 0;
+ null = true;
+ }
flags = (null ? SK_ISNULL : 0) | (indoption[i] << SK_BT_INDOPTION_SHIFT);
ScanKeyEntryInitializeWithInfo(&skey[i],
flags,
@@ -2083,38 +2108,218 @@ btproperty(Oid index_oid, int attno,
}
/*
- * _bt_nonkey_truncate() -- create tuple without non-key suffix attributes.
+ * _bt_suffix_truncate() -- create tuple without unneeded suffix attributes.
*
- * Returns truncated index tuple allocated in caller's memory context, with key
- * attributes copied from caller's itup argument. Currently, suffix truncation
- * is only performed to create pivot tuples in INCLUDE indexes, but some day it
- * could be generalized to remove suffix attributes after the first
- * distinguishing key attribute.
+ * Returns truncated pivot index tuple allocated in caller's memory context,
+ * with key attributes copied from caller's firstright argument. If rel is
+ * an INCLUDE index, non-key attributes will definitely be truncated away,
+ * since they're not part of the key space. More aggressive suffix
+ * truncation can take place when it's clear that the returned tuple does not
+ * need one or more suffix key attributes. This is possible when there are
+ * attributes that follow an attribute in firstright that is not equal to the
+ * corresponding attribute in lastleft (equal according to an insertion scan
+ * key).
*
- * Truncated tuple is guaranteed to be no larger than the original, which is
- * important for staying under the 1/3 of a page restriction on tuple size.
+ * Sometimes this routine will return a new pivot tuple that's larger than
+ * firstright, because a new heap TID attribute had to be added to
+ * distinguish lastleft from firstright. This should only happen when the
+ * caller is in the process of splitting a leaf page that has many logical
+ * duplicates, where it's unavoidable.
*
- * Note that returned tuple's t_tid offset will hold the number of attributes
- * present, so the original item pointer offset is not represented. Caller
- * should only change truncated tuple's downlink.
+ * Note that returned tuple's t_tid offset will hold the number of
+ * attributes present, so the original item pointer offset is not
+ * represented. Caller should only change truncated tuple's downlink. Note
+ * also that truncated key attributes are treated as containing "minus
+ * infinity" values by _bt_compare()/_bt_tuple_compare(). Returned tuple is
+ * guaranteed to be no larger than the original plus some extra space for a
+ * possible extra heap TID tie-breaker attribute, which is important for
+ * staying under the 1/3 of a page restriction on tuple size.
*/
IndexTuple
-_bt_nonkey_truncate(Relation rel, IndexTuple itup)
+_bt_suffix_truncate(Relation rel, IndexTuple lastleft, IndexTuple firstright)
{
- int nkeyattrs = IndexRelationGetNumberOfKeyAttributes(rel);
- IndexTuple truncated;
+ TupleDesc itupdesc = RelationGetDescr(rel);
+ int16 natts = IndexRelationGetNumberOfAttributes(rel);
+ int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
+ int leavenatts;
+ IndexTuple pivot;
+ ItemPointer pivotheaptid;
+ Size newsize;
/*
- * We should only ever truncate leaf index tuples, which must have both
- * key and non-key attributes. It's never okay to truncate a second time.
+ * We should only ever truncate leaf index tuples, which must have non-key
+ * attributes in the case of INCLUDE indexes. It's never okay to truncate
+ * a second time.
+ */
+ Assert(BTreeTupleGetNAtts(lastleft, rel) == natts);
+ Assert(BTreeTupleGetNAtts(firstright, rel) == natts);
+
+ /* Determine how many attributes must be left behind */
+ leavenatts = _bt_leave_natts(rel, lastleft, firstright);
+
+ if (leavenatts <= natts)
+ {
+ IndexTuple tidpivot;
+
+ /*
+ * Truncate away non-key attributes and/or key attributes. Do a
+ * straight copy in the case where the only attribute to be "truncated
+ * away" is the implicit heap TID key attribute (i.e. the case where we
+ * can at least avoid adding an explicit heap TID attribute to new
+ * pivot). We should only call index_truncate_tuple() when user
+ * attributes need to be truncated.
+ */
+ if (leavenatts < natts)
+ pivot = index_truncate_tuple(itupdesc, firstright, leavenatts);
+ else
+ pivot = CopyIndexTuple(firstright);
+
+ /*
+ * If there is a distinguishing key attribute within leavenatts, there
+ * is no need to add an explicit heap TID attribute to new pivot.
+ */
+ if (leavenatts <= nkeyatts)
+ {
+ BTreeTupleSetNAtts(pivot, leavenatts);
+ return pivot;
+ }
+
+ /*
+ * Only non-key attributes could be truncated away from an INCLUDE
+ * index's pivot tuple. They are not considered part of the key space,
+ * so it's still necessary to add a heap TID attribute to the new pivot
+ * tuple. Create enlarged copy of our truncated right tuple copy, to
+ * fit heap TID.
+ */
+ Assert(natts < nkeyatts);
+ newsize = IndexTupleSize(pivot) + MAXALIGN(sizeof(ItemPointerData));
+ tidpivot = palloc0(newsize);
+ memcpy(tidpivot, pivot, IndexTupleSize(pivot));
+ pfree(pivot);
+ pivot = tidpivot;
+ }
+ else
+ {
+ /*
+ * No truncation was possible, since attributes are all equal. It's
+ * necessary to add a heap TID attribute to the new pivot tuple.
+ */
+ Assert(natts == nkeyatts);
+ newsize = IndexTupleSize(firstright) + MAXALIGN(sizeof(ItemPointerData));
+ pivot = palloc0(newsize);
+ memcpy(pivot, firstright, IndexTupleSize(firstright));
+ }
+
+ /*
+ * Create enlarged copy of first right tuple to fit heap TID. We must
+ * use heap TID as a unique-ifier in new pivot tuple, since no user key
+ * attribute distinguishes which values belong on each side of the split
+ * point.
+ */
+ pivot->t_info &= ~INDEX_SIZE_MASK;
+ pivot->t_info |= newsize;
+
+ /*
+ * Generate an artificial heap TID value for the new pivot tuple. This
+ * will be the median of the left and right heap TIDs, or a close
+ * approximation.
+ *
+ * Note that we deliberately pass the firstright heap TID as low and the
+ * lastleft heap TID as high, since the implicit heap TID attribute has
+ * DESC sort order.
+ *
+ * Lehman and Yao require that the downlink to the right page, which is
+ * to be inserted into the parent page in the second phase of a page
+ * split be a strict lower bound on all current and future items on the
+ * right page (this will be copied from the new high key for the left
+ * side of the split). New pivot's heap TID attribute may occasionally
+ * be equal to the the lastleft heap TID, but it must never be equal to
+ * firstright's heap TID.
*/
- Assert(BTreeTupleGetNAtts(itup, rel) ==
- IndexRelationGetNumberOfAttributes(rel));
+ pivotheaptid = (ItemPointer) ((char *) pivot + newsize -
+ MAXALIGN(sizeof(ItemPointerData)));
+ _bt_set_median_tid(&firstright->t_tid, &lastleft->t_tid, pivotheaptid);
+ Assert(ItemPointerCompare(&lastleft->t_tid, pivotheaptid) >= 0);
+ Assert(ItemPointerCompare(&firstright->t_tid, pivotheaptid) < 0);
- truncated = index_truncate_tuple(RelationGetDescr(rel), itup, nkeyattrs);
- BTreeTupleSetNAtts(truncated, nkeyattrs);
+ /* Mark tuple as containing all key attributes, plus TID attribute */
+ BTreeTupleSetNAtts(pivot, nkeyatts);
+ BTreeTupleSetAltHeapTID(pivot);
+
+ return pivot;
+}
+
+/*
+ * _bt_leave_natts - how many key attributes to leave when truncating.
+ *
+ * This can return a number of attributes that is one greater than the
+ * number of key attributes for the index relation. This indicates that the
+ * caller must use a heap TID as a unique-ifier in new pivot tuple.
+ */
+static int
+_bt_leave_natts(Relation rel, IndexTuple lastleft, IndexTuple firstright)
+{
+ int nkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
+ int leavenatts;
+ ScanKey skey;
+
+ skey = _bt_mkscankey(rel, firstright);
+
+ /*
+ * Even test nkeyatts (no truncated user attributes) case, since caller
+ * cares about whether or not it can avoid appending a heap TID as a
+ * unique-ifier
+ */
+ leavenatts = 1;
+ for(;;)
+ {
+ if (leavenatts > nkeyatts)
+ break;
+ if (_bt_tuple_compare(rel, leavenatts, skey, NULL, lastleft) > 0)
+ break;
+ leavenatts++;
+ }
+
+ /* Can't leak memory here */
+ _bt_freeskey(skey);
+
+ return leavenatts;
+}
+
+/*
+ * _bt_set_median_tid - set's item pointer to median TID value.
+ */
+static void
+_bt_set_median_tid(ItemPointer low, ItemPointer high,
+ ItemPointer pivotheaptid)
+{
+ uint64 lowblock, highblock, medianblock;
+ uint32 lowoffset, highoffset, medianoffset;
+
+ Assert(ItemPointerCompare(low, high) < 0);
+
+ lowblock = ItemPointerGetBlockNumber(low);
+ highblock = ItemPointerGetBlockNumber(high);
+
+ lowoffset = ItemPointerGetOffsetNumber(low);
+ highoffset = ItemPointerGetOffsetNumber(high);
+
+ medianblock = (lowblock + highblock) / 2;
+ if (medianblock >= highblock)
+ {
+ medianblock = highblock;
+
+ /* Cannot allow result to equal low */
+ if (medianblock == highblock)
+ medianoffset = highoffset;
+ else
+ medianoffset = (lowoffset + highoffset) / 2;
+ }
+ else
+ medianoffset = Max(lowoffset, highoffset) + 1;
- return truncated;
+ ItemPointerSetBlockNumber(pivotheaptid, medianblock);
+ ItemPointerSetOffsetNumber(pivotheaptid, medianoffset);
}
/*
@@ -2137,6 +2342,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
int16 nkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);
BTPageOpaque opaque = (BTPageOpaque) PageGetSpecialPointer(page);
IndexTuple itup;
+ int tupnatts;
/*
* We cannot reliably test a deleted or half-deleted page, since they have
@@ -2156,6 +2362,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
"BT_N_KEYS_OFFSET_MASK can't fit INDEX_MAX_KEYS");
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
+ tupnatts = BTreeTupleGetNAtts(itup, rel);
if (P_ISLEAF(opaque))
{
@@ -2165,7 +2372,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
* Leaf tuples that are not the page high key (non-pivot tuples)
* should never be truncated
*/
- return BTreeTupleGetNAtts(itup, rel) == natts;
+ return tupnatts == natts;
}
else
{
@@ -2176,7 +2383,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
Assert(!P_RIGHTMOST(opaque));
/* Page high key tuple contains only key attributes */
- return BTreeTupleGetNAtts(itup, rel) == nkeyatts;
+ return tupnatts > 0 && tupnatts <= nkeyatts;
}
}
else /* !P_ISLEAF(opaque) */
@@ -2209,7 +2416,7 @@ _bt_check_natts(Relation rel, Page page, OffsetNumber offnum)
* Tuple contains only key attributes despite on is it page high
* key or not
*/
- return BTreeTupleGetNAtts(itup, rel) == nkeyatts;
+ return tupnatts > 0 && tupnatts <= nkeyatts;
}
}
diff --git a/src/backend/access/nbtree/nbtxlog.c b/src/backend/access/nbtree/nbtxlog.c
index 67a94cb80a..7c061e96d2 100644
--- a/src/backend/access/nbtree/nbtxlog.c
+++ b/src/backend/access/nbtree/nbtxlog.c
@@ -202,7 +202,7 @@ btree_xlog_insert(bool isleaf, bool ismeta, XLogReaderState *record)
}
static void
-btree_xlog_split(bool onleft, bool lhighkey, XLogReaderState *record)
+btree_xlog_split(bool onleft, XLogReaderState *record)
{
XLogRecPtr lsn = record->EndRecPtr;
xl_btree_split *xlrec = (xl_btree_split *) XLogRecGetData(record);
@@ -213,8 +213,6 @@ btree_xlog_split(bool onleft, bool lhighkey, XLogReaderState *record)
BTPageOpaque ropaque;
char *datapos;
Size datalen;
- IndexTuple left_hikey = NULL;
- Size left_hikeysz = 0;
BlockNumber leftsib;
BlockNumber rightsib;
BlockNumber rnext;
@@ -248,20 +246,6 @@ btree_xlog_split(bool onleft, bool lhighkey, XLogReaderState *record)
_bt_restore_page(rpage, datapos, datalen);
- /*
- * When the high key isn't present is the wal record, then we assume it to
- * be equal to the first key on the right page. It must be from the leaf
- * level.
- */
- if (!lhighkey)
- {
- ItemId hiItemId = PageGetItemId(rpage, P_FIRSTDATAKEY(ropaque));
-
- Assert(isleaf);
- left_hikey = (IndexTuple) PageGetItem(rpage, hiItemId);
- left_hikeysz = ItemIdGetLength(hiItemId);
- }
-
PageSetLSN(rpage, lsn);
MarkBufferDirty(rbuf);
@@ -284,6 +268,8 @@ btree_xlog_split(bool onleft, bool lhighkey, XLogReaderState *record)
OffsetNumber off;
IndexTuple newitem = NULL;
Size newitemsz = 0;
+ IndexTuple left_hikey = NULL;
+ Size left_hikeysz = 0;
Page newlpage;
OffsetNumber leftoff;
@@ -298,13 +284,10 @@ btree_xlog_split(bool onleft, bool lhighkey, XLogReaderState *record)
}
/* Extract left hikey and its size (assuming 16-bit alignment) */
- if (lhighkey)
- {
- left_hikey = (IndexTuple) datapos;
- left_hikeysz = MAXALIGN(IndexTupleSize(left_hikey));
- datapos += left_hikeysz;
- datalen -= left_hikeysz;
- }
+ left_hikey = (IndexTuple) datapos;
+ left_hikeysz = MAXALIGN(IndexTupleSize(left_hikey));
+ datapos += left_hikeysz;
+ datalen -= left_hikeysz;
Assert(datalen == 0);
@@ -1003,16 +986,10 @@ btree_redo(XLogReaderState *record)
btree_xlog_insert(false, true, record);
break;
case XLOG_BTREE_SPLIT_L:
- btree_xlog_split(true, false, record);
- break;
- case XLOG_BTREE_SPLIT_L_HIGHKEY:
- btree_xlog_split(true, true, record);
+ btree_xlog_split(true, record);
break;
case XLOG_BTREE_SPLIT_R:
- btree_xlog_split(false, false, record);
- break;
- case XLOG_BTREE_SPLIT_R_HIGHKEY:
- btree_xlog_split(false, true, record);
+ btree_xlog_split(false, record);
break;
case XLOG_BTREE_VACUUM:
btree_xlog_vacuum(record);
diff --git a/src/backend/access/rmgrdesc/nbtdesc.c b/src/backend/access/rmgrdesc/nbtdesc.c
index 5c4457179d..667c906b2e 100644
--- a/src/backend/access/rmgrdesc/nbtdesc.c
+++ b/src/backend/access/rmgrdesc/nbtdesc.c
@@ -35,8 +35,6 @@ btree_desc(StringInfo buf, XLogReaderState *record)
}
case XLOG_BTREE_SPLIT_L:
case XLOG_BTREE_SPLIT_R:
- case XLOG_BTREE_SPLIT_L_HIGHKEY:
- case XLOG_BTREE_SPLIT_R_HIGHKEY:
{
xl_btree_split *xlrec = (xl_btree_split *) rec;
@@ -130,12 +128,6 @@ btree_identify(uint8 info)
case XLOG_BTREE_SPLIT_R:
id = "SPLIT_R";
break;
- case XLOG_BTREE_SPLIT_L_HIGHKEY:
- id = "SPLIT_L_HIGHKEY";
- break;
- case XLOG_BTREE_SPLIT_R_HIGHKEY:
- id = "SPLIT_R_HIGHKEY";
- break;
case XLOG_BTREE_VACUUM:
id = "VACUUM";
break;
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index dfbda5458f..ffeb0624fe 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -854,10 +854,8 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
* PageIndexTupleDelete is the best way. Delete the items in reverse
* order so we don't have to think about adjusting item numbers for
* previous deletions.
- *
- * TODO: tune the magic number here
*/
- if (nitems <= 2)
+ if (nitems <= 7)
{
while (--nitems >= 0)
PageIndexTupleDelete(page, itemnos[nitems]);
diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c
index 9fb33b9035..2a0b64ad47 100644
--- a/src/backend/utils/sort/tuplesort.c
+++ b/src/backend/utils/sort/tuplesort.c
@@ -4057,23 +4057,26 @@ comparetup_index_btree(const SortTuple *a, const SortTuple *b,
}
/*
- * If key values are equal, we sort on ItemPointer. This does not affect
- * validity of the finished index, but it may be useful to have index
- * scans in physical order.
+ * If key values are equal, we sort on ItemPointer. This is required
+ * for btree indexes, since heap TID is treated as an implicit last
+ * key attribute in order to ensure that all keys in the index are
+ * physically unique.
+ *
+ * Deliberately invert the order, since TIDs "sort DESC".
*/
{
BlockNumber blk1 = ItemPointerGetBlockNumber(&tuple1->t_tid);
BlockNumber blk2 = ItemPointerGetBlockNumber(&tuple2->t_tid);
if (blk1 != blk2)
- return (blk1 < blk2) ? -1 : 1;
+ return (blk1 < blk2) ? 1 : -1;
}
{
OffsetNumber pos1 = ItemPointerGetOffsetNumber(&tuple1->t_tid);
OffsetNumber pos2 = ItemPointerGetOffsetNumber(&tuple2->t_tid);
if (pos1 != pos2)
- return (pos1 < pos2) ? -1 : 1;
+ return (pos1 < pos2) ? 1 : -1;
}
return 0;
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 3b9461c3b4..69cafa2195 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -114,16 +114,27 @@ typedef struct BTMetaPageData
#define BTREE_METAPAGE 0 /* first page is meta */
#define BTREE_MAGIC 0x053162 /* magic number of btree pages */
-#define BTREE_VERSION 3 /* current version number */
-#define BTREE_MIN_VERSION 2 /* minimal supported version number */
+/* FIXME: Support versions 2 and 3 for the benefit of pg_upgrade users */
+#define BTREE_VERSION 4 /* current version number */
+#define BTREE_MIN_VERSION 4 /* minimal supported version number */
/*
* Maximum size of a btree index entry, including its tuple header.
*
* We actually need to be able to fit three items on every page,
* so restrict any one item to 1/3 the per-page available space.
+ *
+ * There are rare cases where _bt_suffix_truncate() will need to enlarge
+ * a heap index tuple to make space for a tie-breaker heap TID
+ * attribute, which we account for here.
*/
#define BTMaxItemSize(page) \
+ MAXALIGN_DOWN((PageGetPageSize(page) - \
+ MAXALIGN(SizeOfPageHeaderData + \
+ 3*sizeof(ItemIdData) + \
+ 3*MAXALIGN(sizeof(ItemPointerData))) - \
+ MAXALIGN(sizeof(BTPageOpaqueData))) / 3)
+#define BTMaxItemSizeOld(page) \
MAXALIGN_DOWN((PageGetPageSize(page) - \
MAXALIGN(SizeOfPageHeaderData + 3*sizeof(ItemIdData)) - \
MAXALIGN(sizeof(BTPageOpaqueData))) / 3)
@@ -204,12 +215,11 @@ typedef struct BTMetaPageData
* real offset (downlinks only need to store a block number). The offset
* field only stores the number of attributes when the INDEX_ALT_TID_MASK
* bit is set (we never assume that pivot tuples must explicitly store the
- * number of attributes, and currently do not bother storing the number of
- * attributes unless indnkeyatts actually differs from indnatts).
- * INDEX_ALT_TID_MASK is only used for pivot tuples at present, though it's
- * possible that it will be used within non-pivot tuples in the future. Do
- * not assume that a tuple with INDEX_ALT_TID_MASK set must be a pivot
- * tuple.
+ * number of attributes). INDEX_ALT_TID_MASK is only used for pivot tuples
+ * at present, though it's possible that it will be used within non-pivot
+ * tuples in the future. Do not assume that a tuple with INDEX_ALT_TID_MASK
+ * set must be a pivot tuple. A pivot tuple must have INDEX_ALT_TID_MASK set
+ * as of BTREE_VERSION 4, however.
*
* The 12 least significant offset bits are used to represent the number of
* attributes in INDEX_ALT_TID_MASK tuples, leaving 4 bits that are reserved
@@ -219,6 +229,8 @@ typedef struct BTMetaPageData
#define INDEX_ALT_TID_MASK INDEX_AM_RESERVED_BIT
#define BT_RESERVED_OFFSET_MASK 0xF000
#define BT_N_KEYS_OFFSET_MASK 0x0FFF
+/* Reserved to indicate if heap TID is represented at end of tuple */
+#define BT_HEAP_TID_ATTR 0x1000
/* Get/set downlink block number */
#define BTreeInnerTupleGetDownLink(itup) \
@@ -241,14 +253,15 @@ typedef struct BTMetaPageData
} while(0)
/*
- * Get/set number of attributes within B-tree index tuple. Asserts should be
- * removed when BT_RESERVED_OFFSET_MASK bits will be used.
+ * Get/set number of attributes within B-tree index tuple.
+ *
+ * Note that this does not include an implicit tie-breaker heap-TID
+ * attribute, if any.
*/
#define BTreeTupleGetNAtts(itup, rel) \
( \
(itup)->t_info & INDEX_ALT_TID_MASK ? \
( \
- AssertMacro((ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) & BT_RESERVED_OFFSET_MASK) == 0), \
ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) & BT_N_KEYS_OFFSET_MASK \
) \
: \
@@ -257,10 +270,42 @@ typedef struct BTMetaPageData
#define BTreeTupleSetNAtts(itup, n) \
do { \
(itup)->t_info |= INDEX_ALT_TID_MASK; \
- Assert(((n) & BT_RESERVED_OFFSET_MASK) == 0); \
ItemPointerSetOffsetNumber(&(itup)->t_tid, (n) & BT_N_KEYS_OFFSET_MASK); \
} while(0)
+/*
+ * Get tie-breaker heap TID attribute, if any. Macro works with both pivot
+ * and non-pivot tuples.
+ *
+ * Assumes that any tuple without INDEX_ALT_TID_MASK set has a t_tid that
+ * points to the heap, and that all pivot tuples have INDEX_ALT_TID_MASK set
+ * (since all pivot tuples must as of BTREE_VERSION 4). When non-pivot
+ * tuples use the INDEX_ALT_TID_MASK representation in the future, they'll
+ * probably also contain a heap TID at the end of the tuple. We avoid
+ * assuming that a tuple with INDEX_ALT_TID_MASK set is necessarily a pivot
+ * tuple.
+ */
+#define BTreeTupleGetHeapTID(itup) \
+ ( \
+ (itup)->t_info & INDEX_ALT_TID_MASK && \
+ (ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) & BT_HEAP_TID_ATTR) != 0 ? \
+ ( \
+ (ItemPointer) (((char *) (itup) + IndexTupleSize(itup)) - \
+ MAXALIGN(sizeof(ItemPointerData))) \
+ ) \
+ : (itup)->t_info & INDEX_ALT_TID_MASK ? NULL : (ItemPointer) &((itup)->t_tid) \
+ )
+/*
+ * Set the heap TID attribute for a tuple that uses the INDEX_ALT_TID_MASK
+ * representation (currently limited to pivot tuples)
+ */
+#define BTreeTupleSetAltHeapTID(itup) \
+ do { \
+ Assert((itup)->t_info & INDEX_ALT_TID_MASK); \
+ ItemPointerSetOffsetNumber(&(itup)->t_tid, \
+ ItemPointerGetOffsetNumberNoCheck(&(itup)->t_tid) | BT_HEAP_TID_ATTR); \
+ } while(0)
+
/*
* Operator strategy numbers for B-tree have been moved to access/stratnum.h,
* because many places need to use them in ScanKeyInit() calls.
@@ -564,15 +609,18 @@ extern int _bt_pagedel(Relation rel, Buffer buf);
* prototypes for functions in nbtsearch.c
*/
extern BTStack _bt_search(Relation rel,
- int keysz, ScanKey scankey, bool nextkey,
+ int keysz, ScanKey scankey, ItemPointer scantid, bool nextkey,
Buffer *bufP, int access, Snapshot snapshot);
extern Buffer _bt_moveright(Relation rel, Buffer buf, int keysz,
- ScanKey scankey, bool nextkey, bool forupdate, BTStack stack,
- int access, Snapshot snapshot);
+ ScanKey scankey, ItemPointer scantid, bool nextkey,
+ bool forupdate, BTStack stack, int access, Snapshot snapshot);
extern OffsetNumber _bt_binsrch(Relation rel, Buffer buf, int keysz,
- ScanKey scankey, bool nextkey);
+ ScanKey scankey, ItemPointer scantid, OffsetNumber low,
+ bool nextkey);
extern int32 _bt_compare(Relation rel, int keysz, ScanKey scankey,
- Page page, OffsetNumber offnum);
+ ItemPointer scantid, Page page, OffsetNumber offnum);
+extern int32 _bt_tuple_compare(Relation rel, int keysz, ScanKey scankey,
+ ItemPointer scantid, IndexTuple itup);
extern bool _bt_first(IndexScanDesc scan, ScanDirection dir);
extern bool _bt_next(IndexScanDesc scan, ScanDirection dir);
extern Buffer _bt_get_endpoint(Relation rel, uint32 level, bool rightmost,
@@ -605,7 +653,8 @@ extern bytea *btoptions(Datum reloptions, bool validate);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
-extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup);
+extern IndexTuple _bt_suffix_truncate(Relation rel, IndexTuple lastleft,
+ IndexTuple firstright);
extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum);
/*
diff --git a/src/include/access/nbtxlog.h b/src/include/access/nbtxlog.h
index 819373031c..d0fd4229d2 100644
--- a/src/include/access/nbtxlog.h
+++ b/src/include/access/nbtxlog.h
@@ -28,8 +28,8 @@
#define XLOG_BTREE_INSERT_META 0x20 /* same, plus update metapage */
#define XLOG_BTREE_SPLIT_L 0x30 /* add index tuple with split */
#define XLOG_BTREE_SPLIT_R 0x40 /* as above, new item on right */
-#define XLOG_BTREE_SPLIT_L_HIGHKEY 0x50 /* as above, include truncated highkey */
-#define XLOG_BTREE_SPLIT_R_HIGHKEY 0x60 /* as above, include truncated highkey */
+/* 0x50: unused */
+/* 0x60: unused */
#define XLOG_BTREE_DELETE 0x70 /* delete leaf index tuples for a page */
#define XLOG_BTREE_UNLINK_PAGE 0x80 /* delete a half-dead page */
#define XLOG_BTREE_UNLINK_PAGE_META 0x90 /* same, and update metapage */
@@ -82,20 +82,16 @@ typedef struct xl_btree_insert
*
* Note: the four XLOG_BTREE_SPLIT xl_info codes all use this data record.
* The _L and _R variants indicate whether the inserted tuple went into the
- * left or right split page (and thus, whether newitemoff and the new item
- * are stored or not). The _HIGHKEY variants indicate that we've logged
- * explicitly left page high key value, otherwise redo should use right page
- * leftmost key as a left page high key. _HIGHKEY is specified for internal
- * pages where right page leftmost key is suppressed, and for leaf pages
- * of covering indexes where high key have non-key attributes truncated.
+ * left or right split page (and thus, whether newitemoff and the new item are
+ * stored or not). We always explicitly log the left page high key value.
*
* Backup Blk 0: original page / new left page
*
* The left page's data portion contains the new item, if it's the _L variant.
- * (In the _R variants, the new item is one of the right page's tuples.)
- * If level > 0, an IndexTuple representing the HIKEY of the left page
- * follows. We don't need this on leaf pages, because it's the same as the
- * leftmost key in the new right page.
+ * In the _R variants, the new item is one of the right page's tuples. An
+ * IndexTuple representing the HIKEY of the left page follows. We don't need
+ * this on leaf pages, because it's the same as the leftmost key in the new
+ * right page.
*
* Backup Blk 1: new right page
*
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index dc6262be43..2c20cea4b9 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -5896,8 +5896,8 @@ inner join j1 j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2
where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1;
id1 | id2 | id1 | id2
-----+-----+-----+-----
- 1 | 1 | 1 | 1
1 | 2 | 1 | 2
+ 1 | 1 | 1 | 1
(2 rows)
reset enable_nestloop;
--
2.17.1
0004-Retail-IndexTuple-Deletion-with-TID-sorting-in-Leaf.patch (text/x-patch, 6.7K) download
From 223ba9983e713de76384c06e25f927000e85541a Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 15 Aug 2018 13:35:48 +0500
Subject: [PATCH 4/5] Retail-IndexTuple-Deletion-with-TID-sorting-in-Leaf
---
src/backend/access/nbtree/nbtree.c | 75 ++++++++++++++++++++----------
src/backend/commands/vacuumlazy.c | 8 ++--
src/include/access/genam.h | 2 +-
3 files changed, 55 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 44f4d0dc9f..50378e0f51 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -889,15 +889,19 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
-static int
-tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
-{
- for (int i = 0; i < ntid; i++)
- if (ItemPointerEquals(tid, &(tid_list[i])))
- return i;
- return -1;
-}
-
+/*
+ * Deletion of index entries pointing to heap tuples.
+ *
+ * Constraints:
+ * 1. TID list info->dead_tuples arranged in ASC order.
+ * 2. Logical duplicates of index tuples stored in DESC order.
+ *
+ * The function generates an insertion scan key and descent by btree for first
+ * index tuple what satisfies scan key and last TID in info->dead_tuples list.
+ * For the scan results it deletes all index entries, matched to the TID list.
+ *
+ * Result: a palloc'd struct containing statistical info.
+ */
IndexTargetDeleteResult*
bttargetdelete(IndexTargetDeleteInfo *info,
IndexTargetDeleteResult *stats,
@@ -916,33 +920,33 @@ bttargetdelete(IndexTargetDeleteInfo *info,
int ndeletable = 0;
OffsetNumber deletable[MaxOffsetNumber];
IndexTuple itup;
+ int pos = info->last_dead_tuple;
if (stats == NULL)
stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+ /* Assemble scankey */
itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
skey = _bt_mkscankey(irel, itup);
/* Descend the tree and position ourselves on the target leaf page. */
- stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
- _bt_freestack(stack);
+ stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_READ, NULL);
/* To prepare tuple entries search across index pages */
Assert(BufferIsValid(buf));
- offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], P_FIRSTDATAKEY(opaque), false);
for (;;)
{
int32 cmpval;
ItemId itemid;
IndexTuple itup;
- int pos;
/* Switch to the next page */
- if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ if (offnum > PageGetMaxOffsetNumber(page))
{
/*
* Before unlocking index page we need to delete
@@ -967,20 +971,27 @@ bttargetdelete(IndexTargetDeleteInfo *info,
break;
/*
- * Switch to the next index page
+ * Traverse to a next reliable index page
*/
- buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_READ, NULL);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
- offnum = P_FIRSTDATAKEY(opaque);
- continue;
+ Assert(!P_IGNORE(opaque));
+ /* Set offnum to first potentially interesting item */
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], P_FIRSTDATAKEY(opaque), false);
+
+ if (offnum > PageGetMaxOffsetNumber(page))
+ break;
+ else
+ continue;
}
/*
* This index entry satisfied to the scan key?
*/
- cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+ cmpval = _bt_compare(irel, keysCount, skey, NULL, page, offnum);
if (cmpval != 0)
/* End of index entries, satisfied to the scan key */
@@ -992,15 +1003,28 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
itemid = PageGetItemId(page, offnum);
itup = (IndexTuple) PageGetItem(page, itemid);
- pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
- if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ /*
+ * Search for next TID from presorted btree result comparable
+ * to TID from presorted dead_tuples tid list
+ */
+ while (pos >= 0)
{
- /* index entry for TID of dead tuple is found */
- deletable[ndeletable++] = offnum;
- info->found_dead_tuples[pos] = true;
+ int res = ItemPointerCompare(&(itup->t_tid), &info->dead_tuples[pos]);
+ if ((res == 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+ else if (res > 0)
+ break;
+ pos--;
}
+ if (pos < 0)
+ break;
+
offnum = OffsetNumberNext(offnum);
}
@@ -1017,7 +1041,8 @@ bttargetdelete(IndexTargetDeleteInfo *info,
stats->tuples_removed += ndeletable;
}
- /* Release scan key, unpin and unlock buffer */
+ /* Release stack, scan key, unpin and unlock buffer */
+ _bt_freestack(stack);
_bt_freeskey(skey);
_bt_relbuf(irel, buf);
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 9a625b8dba..1b952fedd9 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -1770,7 +1770,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
econtext->ecxt_scantuple = slot;
/* Get tuple from heap */
- for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1814,9 +1814,9 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
- ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples - tnum;
- ivinfo.found_dead_tuples = found + tnum;
+ ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.last_dead_tuple = tnum;
+ ivinfo.found_dead_tuples = found;
index_target_delete(&ivinfo, &stats, values, isnull);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 79b386eebd..297d357a7d 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -55,7 +55,7 @@ typedef struct IndexTargetDeleteInfo
{
Relation heapRelation;
Relation indexRelation; /* the index being vacuumed */
- int num_dead_tuples;
+ int last_dead_tuple;
ItemPointer dead_tuples;
bool* found_dead_tuples;
} IndexTargetDeleteInfo;
--
2.17.1
0005-Background-Worker.patch (text/x-patch, 113.1K) download
From 3b21afe82bac9098249fb10bf3c3144fb01d871b Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 15 Aug 2018 13:45:22 +0500
Subject: [PATCH 5/5] Background Worker
---
.../postgres_fdw/expected/postgres_fdw.out | 30 +-
contrib/postgres_fdw/sql/postgres_fdw.sql | 4 +-
src/backend/access/heap/pruneheap.c | 5 +-
src/backend/access/nbtree/nbtinsert.c | 19 +
src/backend/access/nbtree/nbtree.c | 23 +-
src/backend/access/transam/xact.c | 4 +
src/backend/catalog/system_views.sql | 10 +
src/backend/commands/vacuumlazy.c | 46 +-
src/backend/postmaster/Makefile | 2 +-
src/backend/postmaster/bgheap.c | 1944 +++++++++++++++++
src/backend/postmaster/pgstat.c | 36 +
src/backend/postmaster/postmaster.c | 160 +-
src/backend/storage/buffer/bufmgr.c | 60 +-
src/backend/storage/buffer/localbuf.c | 24 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/storage/lmgr/proc.c | 5 +-
src/backend/tcop/postgres.c | 12 +
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/backend/utils/hash/Makefile | 2 +-
src/backend/utils/hash/shash.c | 265 +++
src/backend/utils/init/miscinit.c | 3 +-
src/backend/utils/init/postinit.c | 11 +-
src/include/commands/progress.h | 8 +
src/include/commands/vacuum.h | 3 +
src/include/pgstat.h | 14 +-
src/include/postmaster/bgheap.h | 36 +
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 5 +-
src/include/storage/pmsignal.h | 2 +
src/include/utils/shash.h | 54 +
src/test/regress/expected/rules.out | 14 +-
src/test/regress/expected/triggers.out | 2 +-
src/test/regress/input/constraints.source | 12 +-
src/test/regress/output/constraints.source | 34 +-
src/test/regress/sql/rules.sql | 2 +-
src/test/regress/sql/triggers.sql | 2 +-
37 files changed, 2778 insertions(+), 82 deletions(-)
create mode 100644 src/backend/postmaster/bgheap.c
create mode 100644 src/backend/utils/hash/shash.c
create mode 100644 src/include/postmaster/bgheap.h
create mode 100644 src/include/utils/shash.h
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index f5498c62bd..622bea2a7d 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -5496,19 +5496,25 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
- QUERY PLAN
-----------------------------------------------------------------------------------------------------------
- Update on public.ft2
- Output: c1, c2, c3, c4, c5, c6, c7, c8
- Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
- -> Foreign Scan on public.ft2
- Output: c1, c2, NULL::integer, 'bar'::text, c4, c5, c6, c7, c8, ctid
- Filter: (postgres_fdw_abs(ft2.c1) > 2000)
- Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
-(7 rows)
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------
+ Sort
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+ Sort Key: updated.c1
+ CTE updated
+ -> Update on public.ft2
+ Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: ft2.c1, ft2.c2, NULL::integer, 'bar'::text, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.ctid
+ Filter: (postgres_fdw_abs(ft2.c1) > 2000)
+ Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
+ -> CTE Scan on updated
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+(13 rows)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
------+----+-----+----+----+----+------------+----
2001 | 1 | bar | | | | ft2 |
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index e1b955f3f0..6034e83108 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1137,8 +1137,8 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c3 = 'baz'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index ebbafe3275..421dd607bb 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -22,6 +22,7 @@
#include "catalog/catalog.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
@@ -156,6 +157,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/* OK to prune */
(void) heap_page_prune(relation, buffer, OldestXmin, true, &ignore);
+ HeapCleanerSend(relation, BufferGetBlockNumber(buffer));
}
/* And release buffer lock */
@@ -411,7 +413,6 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
- ndeleted++;
}
/* Nothing more to do */
@@ -583,9 +584,9 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- ndeleted++;
if (chainitems[i] == latestdead)
continue;
+ ndeleted++;
heap_prune_record_unused(prstate, chainitems[i]);
}
diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c
index 7d0556b91d..c4cd941610 100644
--- a/src/backend/access/nbtree/nbtinsert.c
+++ b/src/backend/access/nbtree/nbtinsert.c
@@ -493,6 +493,11 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
if (heap_hot_search(&htid, heapRel, SnapshotSelf, NULL))
{
/* Normal case --- it's still live */
+ elog(LOG, "It's still alive. off=%d BLK:(%d, %d)",
+ offset,
+ ItemPointerGetBlockNumber(&(itup->t_tid)),
+ ItemPointerGetOffsetNumber(&(itup->t_tid))
+ );
}
else
{
@@ -2522,7 +2527,21 @@ _bt_vacuum_one_page(Relation rel, Buffer buffer, Relation heapRel)
ItemId itemId = PageGetItemId(page, offnum);
if (ItemIdIsDead(itemId))
+ {
deletable[ndeletable++] = offnum;
+/* elog(LOG, "BACKEND remove (%d, %d) from index relation.",
+ BufferGetBlockNumber(buffer),
+ offnum
+ );
+ if (ItemIdHasStorage(itemId))
+ {
+ IndexTuple itup1 = (IndexTuple) PageGetItem(page, itemId);
+ elog(LOG, "-> HEAP tid: (%d, %d)",
+ ItemPointerGetBlockNumber(&(itup1->t_tid)),
+ ItemPointerGetOffsetNumber(&(itup1->t_tid))
+ );
+ }*/
+ }
}
if (ndeletable > 0)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 50378e0f51..a460cdfba6 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -954,6 +954,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
@@ -1018,7 +1019,27 @@ bttargetdelete(IndexTargetDeleteInfo *info,
info->found_dead_tuples[pos] = true;
}
else if (res > 0)
+ {
+ if (pos == info->last_dead_tuple)
+ elog(LOG, "ByBreak");
break;
+ }/* else
+ if (pos == info->last_dead_tuple)
+ {
+ OffsetNumber off1 = offnum-1;
+ ItemId itemid1 = PageGetItemId(page, off1);
+ IndexTuple itup1 = (IndexTuple) PageGetItem(page, itemid1);
+
+ elog(LOG, "NEQ: off=%d/%d here: (%d, %d) search for: (%d, %d) pre: (%d, %d)", offnum,
+ PageGetMaxOffsetNumber(page),
+ ItemPointerGetBlockNumber(&(itup->t_tid)),
+ ItemPointerGetOffsetNumber(&(itup->t_tid)),
+ ItemPointerGetBlockNumber(&info->dead_tuples[pos]),
+ ItemPointerGetOffsetNumber(&info->dead_tuples[pos]),
+ ItemPointerGetBlockNumber(&(itup1->t_tid)),
+ ItemPointerGetOffsetNumber(&(itup1->t_tid))
+ );
+ }*/
pos--;
}
@@ -1033,11 +1054,11 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
_bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
-
stats->tuples_removed += ndeletable;
}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9aa63c8792..f5e70ae16d 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -40,6 +40,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -2126,6 +2127,7 @@ CommitTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(true);
AtEOXact_Snapshot(true, false);
AtEOXact_ApplyLauncher(true);
@@ -2404,6 +2406,7 @@ PrepareTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
/* don't call AtEOXact_PgStat here; we fixed pgstat state above */
AtEOXact_Snapshot(true, true);
pgstat_report_xact_timestamp(0);
@@ -2606,6 +2609,7 @@ AbortTransaction(void)
AtEOXact_Files(false);
AtEOXact_ComboCid();
AtEOXact_HashTables(false);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(false);
AtEOXact_ApplyLauncher(false);
pgstat_report_xact_timestamp(0);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 7251552419..17623b0f8d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -903,6 +903,16 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_cleaner AS
+ SELECT
+ S.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid,
+ S.param1 AS relations, S.param2 AS queue_len, S.param3 AS false_hits,
+ S.param4 AS cleaned_tuples, S.param5 AS vainly_cleaned_tuples,
+ S.param6 AS missed_tuples
+ FROM pg_stat_get_progress_info('CLEANER') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid;
+
CREATE VIEW pg_user_mappings AS
SELECT
U.oid AS umid,
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 1b952fedd9..8620c07a2b 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -158,8 +158,6 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
-static void quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -742,7 +740,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1394,7 +1392,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1730,7 +1728,12 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
{
offnum = ItemIdGetRedirect(lp);
lp = PageGetItemId(page, offnum);
- Assert(ItemIdIsUsed(lp));
+
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
}
/* Form a tuple */
@@ -1747,30 +1750,34 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
* quick_vacuum_index() -- quick vacuum one index relation.
*
* Delete all the index entries pointing to tuples listed in
- * vacrelstats->dead_tuples.
+ * dead_tuples.
*/
-static void
+void
quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats)
+ ItemPointer dead_tuples,
+ int num_dead_tuples)
{
int tnum;
- bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
- IndexInfo* indexInfo = BuildIndexInfo(irel);
- EState* estate = CreateExecutorState();
- ExprContext* econtext = GetPerTupleExprContext(estate);
- ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
- TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ bool *found = palloc0(num_dead_tuples*sizeof(bool));
+ IndexInfo *indexInfo = BuildIndexInfo(irel);
+ EState *estate = CreateExecutorState();
+ ExprContext *econtext = GetPerTupleExprContext(estate);
+ ExprState *predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
IndexTargetDeleteResult stats;
IndexTargetDeleteInfo ivinfo;
+ Assert(found != NULL);
+
+ stats.tuples_removed = 0;
ivinfo.indexRelation = irel;
ivinfo.heapRelation = hrel;
econtext->ecxt_scantuple = slot;
/* Get tuple from heap */
- for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
+ for (tnum = num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1781,12 +1788,13 @@ quick_vacuum_index(Relation irel, Relation hrel,
continue;
/* Get a tuple from heap */
- if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ if ((tuple = get_tuple_by_tid(hrel, &(dead_tuples[tnum]))) == NULL)
{
/*
* Tuple has 'not used' status.
*/
found[tnum] = true;
+ elog(LOG, "NULL");
continue;
}
@@ -1805,6 +1813,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
{
found[tnum] = true;
+ elog(LOG, "Predicate");
continue;
}
@@ -1814,15 +1823,16 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.dead_tuples = dead_tuples;
ivinfo.last_dead_tuple = tnum;
ivinfo.found_dead_tuples = found;
+
index_target_delete(&ivinfo, &stats, values, isnull);
}
- pfree(found);
ExecDropSingleTupleTableSlot(slot);
FreeExecutorState(estate);
+ pfree(found);
}
/*
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c23211b2..11796375e8 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
+OBJS = autovacuum.o bgheap.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgheap.c b/src/backend/postmaster/bgheap.c
new file mode 100644
index 0000000000..d49f294d74
--- /dev/null
+++ b/src/backend/postmaster/bgheap.c
@@ -0,0 +1,1944 @@
+/*
+ * bgheap.c
+ *
+ * PostgreSQL integrated cleaner of HEAP and INDEX relations
+ *
+ * Made in autovacuum analogy. Uses 'Target' strategy for clean relations,
+ * without full scan.
+ * The cleaner to consist of one Launcher and Workers.
+ * One worker corresponds to one database.
+ * Launcher receives message from a backend {dbOid; relOid; blkno} by
+ * socket and translate it to a worker by shared memory buffer. Worker receive
+ * message and cleanup block of heap relation and its index relations.
+ *
+ * Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/bgheap.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <unistd.h>
+#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+#include "access/genam.h"
+#include "access/heapam.h"
+#include "access/heapam_xlog.h"
+#include "access/htup_details.h"
+#include "access/nbtree.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/pg_database.h"
+#include "catalog/pg_namespace.h"
+#include "catalog/index.h"
+#include "catalog/namespace.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
+#include "common/ip.h"
+#include "executor/executor.h"
+#include "libpq/pqsignal.h"
+#include "postmaster/bgheap.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "storage/condition_variable.h"
+#include "storage/fd.h"
+#include "storage/freespace.h"
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/pmsignal.h"
+#include "storage/procarray.h"
+#include "storage/smgr.h"
+#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
+#include "utils/lsyscache.h"
+#include "utils/resowner.h"
+#include "utils/shash.h"
+#include "utils/syscache.h"
+#include "utils/timeout.h"
+
+/*
+ * Maximum number of task items in storage at a backend side before shipping to a
+ * background heap cleaner
+ */
+#define BACKEND_DIRTY_ITEMS_MAX (20)
+
+/*
+ * Maximum number of task items in waiting list at a Launcher side.
+ * It is necessary to make it sufficiently large. If a number of arrived
+ * messages exceed this, we ignore the remains and increase counter of missed
+ * items.
+ */
+#define WAITING_MESSAGES_MAX_NUM (100)
+
+/*
+ * Maximum number of dirty blocks which can keep a worker for each relation
+ */
+#define WORKER_DIRTYBLOCKS_MAX_NUM (1000)
+
+/* Maximum number of slots for dirty relations */
+#define WORKER_RELATIONS_MAX_NUM (100)
+
+/* Maximum number of task items in a launcher/worker shared buffer */
+#define WORKER_TASK_ITEMS_MAX (1000)
+
+/*
+ * Maximum time interval which worker can idle without a task (ms)
+ */
+#define WORKER_IDLE_TIME_DURATION_MAX (5000)
+
+/* Minimal info for cleanup a block of heap relation and its index relations */
+typedef struct CleanerMessage
+{
+ Oid dbNode; /* Database ID */
+ Oid relid; /* Relation ID */
+ int blkno; /* Block number */
+ TransactionId xid;
+ uint16 hits;
+} CleanerMessage;
+
+typedef struct WorkerTask
+{
+ BlockNumber blkno;
+ TransactionId lastXid;
+ uint16 hits;
+} WorkerTask;
+
+/*
+ * Shared memory data to control and two-way communication with worker
+ */
+typedef struct WorkerInfoData
+{
+ dlist_node links;
+ Oid dbOid; /* Database ID of the worker */
+ TimestampTz launchtime; /* To define a time of last worker activity */
+ int pid; /* Used for system signals passing */
+ CleanerMessage buffer[WORKER_TASK_ITEMS_MAX]; /* Array of work items */
+ int nitems; /* Number of work items in buffer */
+ LWLock WorkItemLock; /* Locker for safe buffer access */
+ int id; /* Used for Internal launcher buffers management */
+} WorkerInfoData;
+
+typedef struct WorkerInfoData *WorkerInfo;
+
+/*
+ * Shared memory lists.
+ * Launcher get an element from freeWorkers list and init startingWorker value.
+ * Worker set startingWorker to NULL value after startup and add himself
+ * to runningWorkers list.
+ */
+typedef struct HeapCleanerShmemStruct
+{
+ dlist_head freeWorkers;
+ dlist_head runningWorkers;
+ WorkerInfo startingWorker;
+} HeapCleanerShmemStruct;
+
+typedef struct DirtyRelation
+{
+ Oid relid;
+ SHTAB *items;
+} DirtyRelation;
+
+static int stat_vainly_cleaned_tuples = 0;
+static uint64 stat_total_deletions = 0;
+static int stat_false_block_hits = 0;
+static uint32 stat_missed_blocks = 0;
+
+static MemoryContext BGHeapMemCxt = NULL;
+
+/*
+ * Table of database relations.
+ * For each relation we support a waiting list of dirty blocks.
+ */
+static PSHTAB PrivateRelationsTable = NULL;
+
+static bool am_heapcleaner_launcher = false;
+static bool am_heapcleaner_worker = false;
+
+int heapcleaner_max_workers = 10;
+static WorkerInfo MyWorkerInfo = NULL;
+static HeapCleanerShmemStruct *HeapCleanerShmem;
+
+/*
+ * Hash table for collection of dirty blocks after heap_page_prune() action
+ * at a Backend side.
+ */
+PSHTAB dblocks = NULL;
+
+/* Signal handling */
+static volatile sig_atomic_t got_SIGHUP = false;
+static volatile sig_atomic_t got_SIGTERM = false;
+static volatile sig_atomic_t got_SIGUSR2 = false;
+
+NON_EXEC_STATIC pgsocket HeapCleanerSock = PGINVALID_SOCKET;
+static struct sockaddr_storage HeapCleanerSockAddr;
+
+static int TrancheId;
+
+/*
+ * Hash table for waiting data: one for each active worker and one for all messages
+ * intended to not-running workers. Reasons:
+ * 1. Worker in startup process.
+ * 2. Another worker in startup and we can't launch new worker.
+ * 3. We have not free slots for new workers.
+ */
+PSHTAB *wTab;
+
+/*
+ * Parameters of each hash table
+ */
+SHTABCTL wTabCtl;
+
+#ifdef EXEC_BACKEND
+static pid_t hclauncher_forkexec(void);
+static pid_t hcworker_forkexec(void);
+#endif
+
+static uint64 CleanerMessageHashFunc(void *key, uint64 size, uint64 base);
+static PSHTAB cleanup_relations(DirtyRelation *res, SHTAB *AuxiliaryList, bool got_SIGTERM);
+static bool DefaultCompareFunc(void* bucket1, void* bucket2);
+NON_EXEC_STATIC void HeapCleanerLauncherMain(int argc, char *argv[]) pg_attribute_noreturn();
+NON_EXEC_STATIC void HeapCleanerWorkerMain(int argc, char *argv[]) pg_attribute_noreturn();
+static bool isEqualMsgs(void *arg1, void *arg2);
+
+static void launch_worker(Oid dbNode);
+static WorkerInfo look_for_worker(Oid dbNode);
+static void main_launcher_loop(void);
+static void main_worker_loop(void);
+
+static void SIGHUP_Handler(SIGNAL_ARGS);
+static void SIGTERM_Handler(SIGNAL_ARGS);
+static void SIGUSR2_Handler(SIGNAL_ARGS);
+static void backend_send_dirty_blocks(void);
+
+#ifdef EXEC_BACKEND
+/*
+ * forkexec routine for the autovacuum launcher process.
+ *
+ * Format up the arglist, then fork and exec.
+ */
+static pid_t
+hclauncher_forkexec(void)
+{
+ char *av[10];
+ int ac = 0;
+
+ av[ac++] = "postgres";
+ av[ac++] = "--forkhclauncher";
+ av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ av[ac] = NULL;
+
+ Assert(ac < lengthof(av));
+
+ return postmaster_forkexec(ac, av);
+}
+
+/*
+ * We need this set from the outside, before InitProcess is called
+ */
+void
+HeapCleanerLauncherIAm(void)
+{
+ am_heapcleaner_launcher = true;
+}
+
+static pid_t
+hcworker_forkexec(void)
+{
+ char *av[10];
+ int ac = 0;
+
+ av[ac++] = "postgres";
+ av[ac++] = "--forkhcworker";
+ av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ av[ac] = NULL;
+
+ Assert(ac < lengthof(av));
+
+ return postmaster_forkexec(ac, av);
+}
+
+/*
+ * We need this set from the outside, before InitProcess is called
+ */
+void
+HeapCleanerWorkerIAm(void)
+{
+ am_heapcleaner_worker = true;
+}
+#endif
+
+/*
+ * Send dirty blocks from a collector's table to launcher at the end of transaction
+ */
+void
+AtEOXact_BGHeap_tables(bool isCommit)
+{
+ MemoryContext oldMemCxt;
+
+ if (BGHeapMemCxt == NULL)
+ BGHeapMemCxt = AllocSetContextCreate(BGHeapMemCxt, "bgheap",
+ ALLOCSET_DEFAULT_SIZES);
+ oldMemCxt = MemoryContextSwitchTo(BGHeapMemCxt);
+
+ backend_send_dirty_blocks();
+
+ MemoryContextSwitchTo(oldMemCxt);
+}
+
+static void
+save_to_list(PSHTAB AuxiliaryList, WorkerTask *item)
+{
+ bool found;
+
+ WorkerTask *new_item = (WorkerTask *)
+ SHASH_Search(AuxiliaryList,
+ (void *) &(item->blkno),
+ HASH_ENTER, &found);
+
+ Assert((new_item != NULL) && (!found));
+ new_item->hits = item->hits;
+ new_item->lastXid = item->lastXid;
+}
+
+/*
+ * Main logic of HEAP and index relations cleaning
+ */
+static PSHTAB
+cleanup_relations(DirtyRelation *res, PSHTAB AuxiliaryList, bool got_SIGTERM)
+{
+ Relation heapRelation;
+ Relation *IndexRelations;
+ int nindexes;
+ LOCKMODE lockmode = ExclusiveLock/*AccessShareLock*/;
+ WorkerTask *item;
+
+ Assert(res != NULL);
+ Assert(res->items != NULL);
+ Assert(AuxiliaryList != NULL);
+ Assert(SHASH_Entries(AuxiliaryList) == 0);
+
+ if (SHASH_Entries(res->items) == 0)
+ return AuxiliaryList;
+
+ if (RecoveryInProgress())
+ {
+ SHASH_Clean(res->items);
+ return AuxiliaryList;
+ }
+
+ CHECK_FOR_INTERRUPTS();
+
+ StartTransactionCommand();
+
+ /*
+ * At this point relation availability is not guaranteed.
+ * Make safe test to check this.
+ */
+ heapRelation = try_relation_open(res->relid, lockmode);
+
+ if (!heapRelation)
+ {
+ elog(LOG, "[%d] Cleanup: UnSuccessful opening. Relation deleted? dbOid=%d", res->relid, MyWorkerInfo->dbOid);
+ CommitTransactionCommand();
+ SHASH_Clean(res->items);
+ return AuxiliaryList;
+ }
+
+ /*
+ * Check relation type similarly vacuum
+ */
+ if (!(pg_class_ownercheck(RelationGetRelid(heapRelation), GetUserId()) ||
+ (pg_database_ownercheck(MyDatabaseId, GetUserId()) && !heapRelation->rd_rel->relisshared)))
+ {
+ if (heapRelation->rd_rel->relisshared)
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- only superuser can vacuum it",
+ RelationGetRelationName(heapRelation))));
+ else if (heapRelation->rd_rel->relnamespace == PG_CATALOG_NAMESPACE)
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- only superuser or database owner can vacuum it",
+ RelationGetRelationName(heapRelation))));
+ else
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- only table or database owner can vacuum it",
+ RelationGetRelationName(heapRelation))));
+ relation_close(heapRelation, lockmode);
+ CommitTransactionCommand();
+ SHASH_Clean(res->items);
+ return AuxiliaryList;
+ }
+
+ /* Open and lock index relations correspond to the heap relation */
+ vac_open_indexes(heapRelation, RowExclusiveLock, &nindexes, &IndexRelations);
+
+ /* Main cleanup cycle */
+ for (SHASH_SeqReset(res->items);
+ (item = (WorkerTask *) SHASH_SeqNext(res->items)) != NULL; )
+ {
+ BlockNumber nblocks;
+ bool needLock;
+ Buffer buffer;
+ ItemPointerData dead_tuples[MaxOffsetNumber];
+ int num_dead_tuples = 0;
+ OffsetNumber offnum;
+ int irnum;
+ Page page;
+ ItemId lp;
+ int tnum;
+ bool found_non_nbtree = false;
+
+ Assert(item->hits > 0);
+
+ if (!got_SIGTERM &&
+ (res->items->Header.ElementsMaxNum/SHASH_Entries(res->items) > 3) &&
+ (item->lastXid > GetOldestXmin(NULL, PROCARRAY_FLAGS_VACUUM)) &&
+ (item->hits < 10))
+ {
+ /*
+ * Skip block cleaning and save it to a waiting list
+ * for the next iteration.
+ */
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+
+ needLock = !RELATION_IS_LOCAL(heapRelation);
+ if (needLock)
+ LockRelationForExtension(heapRelation, ExclusiveLock);
+ nblocks = RelationGetNumberOfBlocks(heapRelation);
+ if (needLock)
+ UnlockRelationForExtension(heapRelation, ExclusiveLock);
+
+ if (item->blkno >= nblocks)
+ {
+ /*
+ * Block was deleted early.
+ * Skip cleaning and drop block from waiting list.
+ * Accumulate some stats before continue.
+ */
+ stat_false_block_hits += item->hits;
+ pgstat_progress_update_param(PROGRESS_CLEANER_FALSE_HITS, stat_false_block_hits);
+ continue;
+ }
+
+ /* Create TID list */
+ if (!got_SIGTERM)
+ buffer = ReadBufferExtended(heapRelation, MAIN_FORKNUM, item->blkno, RBM_NORMAL_NO_READ, NULL);
+ else
+ buffer = ReadBuffer(heapRelation, item->blkno);
+
+ if (BufferIsInvalid(buffer))
+ {
+ /*
+ * Buffer was already evicted from shared buffers
+ */
+ Assert(!got_SIGTERM);
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+
+ if (!ConditionalLockBuffer(buffer))
+ {
+ /* Can't lock buffer. */
+ ReleaseBuffer(buffer);
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+
+ if (!IsBufferDirty(buffer))
+ {
+ /* Skip block if it is not dirty */
+ UnlockReleaseBuffer(buffer);
+ continue;
+ }
+
+ page = BufferGetPage(buffer);
+
+ /* Collect dead tuples TID's */
+ for (offnum = FirstOffsetNumber;
+ offnum < PageGetMaxOffsetNumber(page);
+ offnum = OffsetNumberNext(offnum))
+ {
+ lp = PageGetItemId(page, offnum);
+ if (ItemIdIsDead(lp)/* && ItemIdHasStorage(lp)*/)
+ {
+ ItemPointerSet(&(dead_tuples[num_dead_tuples]), item->blkno, offnum);
+ num_dead_tuples++;
+ }
+ }
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+
+ if (num_dead_tuples == 0)
+ {
+ /*
+ * Block is clear. Accumulate some stats and continue.
+ */
+ stat_vainly_cleaned_tuples += item->hits;
+ pgstat_progress_update_param(PROGRESS_CLEANER_VAINLY_CLEANED_TUPLES, stat_vainly_cleaned_tuples);
+ ReleaseBuffer(buffer);
+ continue;
+ }
+
+ /* Iterate across all index relations */
+ for (irnum = 0; irnum < nindexes; irnum++)
+ {
+ if (IndexRelations[irnum]->rd_amroutine->amtargetdelete == NULL)
+ {
+ /*
+ * Can clean only btree indexes now.
+ */
+ found_non_nbtree = true;
+ continue;
+ }
+
+ quick_vacuum_index(IndexRelations[irnum],
+ heapRelation,
+ dead_tuples,
+ num_dead_tuples);
+ }
+
+ /*
+ * If heap relation has not only b-tree indexes, can't clean heap block.
+ */
+ if (!found_non_nbtree)
+ {
+ OffsetNumber unusable[MaxOffsetNumber];
+ int nunusable = 0;
+ Size freespace;
+
+ if (!ConditionalLockBufferForCleanup(buffer))
+ {
+ ReleaseBuffer(buffer);
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+
+ START_CRIT_SECTION();
+
+ /* Release DEAD heap tuples storage */
+ for (tnum = 0; tnum < num_dead_tuples; tnum++)
+ {
+ OffsetNumber offnum = ItemPointerGetOffsetNumber(&dead_tuples[tnum]);
+ ItemId lp = PageGetItemId(page, offnum);
+
+ Assert(ItemIdIsDead(lp));
+ ItemIdSetUnused(lp);
+ unusable[nunusable++] = offnum;
+ }
+
+ if (nunusable > 0)
+ {
+ PageRepairFragmentation(page);
+ PageClearFull(page);
+ MarkBufferDirty(buffer);
+
+ if (RelationNeedsWAL(heapRelation))
+ {
+ XLogRecPtr recptr = log_heap_clean(heapRelation, buffer,
+ NULL, 0,
+ NULL, 0,
+ unusable, nunusable,
+ InvalidTransactionId);
+
+ PageSetLSN(BufferGetPage(buffer), recptr);
+ }
+ stat_total_deletions += nunusable;
+ pgstat_progress_update_param(PROGRESS_CLEANER_TOTAL_DELETIONS, stat_total_deletions);
+ }
+
+ END_CRIT_SECTION();
+ freespace = PageGetHeapFreeSpace(page);
+
+ UnlockReleaseBuffer(buffer);
+ RecordPageWithFreeSpace(heapRelation, item->blkno, freespace);
+ pgstat_update_heap_dead_tuples(heapRelation, nunusable);
+ }
+ /*
+ * ToDo: that we will do with TOAST relation?
+ */
+ }
+
+ vac_close_indexes(nindexes, IndexRelations, RowExclusiveLock/*NoLock*/);
+ relation_close(heapRelation, lockmode);
+ CommitTransactionCommand();
+
+ {
+ SHTAB *tmp = res->items;
+ SHASH_Clean(tmp);
+ res->items = AuxiliaryList;
+ return tmp;
+ }
+}
+
+static bool
+DefaultCompareFunc(void* bucket1, void* bucket2)
+{
+ WorkerTask *item1 = (WorkerTask *) bucket1;
+ WorkerTask *item2 = (WorkerTask *) bucket2;
+
+ if (item1->blkno == item2->blkno)
+ return true;
+ else
+ return false;
+}
+
+/*
+ * Free any launcher resources before close process
+ */
+static void
+FreeLauncherInfo(int code, Datum arg)
+{
+ int counter;
+
+ for (counter = 0; counter < heapcleaner_max_workers+1; counter++)
+ SHASH_Destroy(wTab[counter]);
+ pfree(wTab);
+}
+
+/*
+ * Return a WorkerInfo to the free list
+ */
+static void
+FreeWorkerInfo(int code, Datum arg)
+{
+ if (MyWorkerInfo != NULL)
+ {
+ LWLockAcquire(HeapCleanerLock, LW_EXCLUSIVE);
+
+ dlist_delete(&MyWorkerInfo->links);
+ MyWorkerInfo->dbOid = InvalidOid;
+ MyWorkerInfo->launchtime = 0;
+ dlist_push_head(&HeapCleanerShmem->freeWorkers,
+ &MyWorkerInfo->links);
+ /* not mine anymore */
+ MyWorkerInfo = NULL;
+
+ LWLockRelease(HeapCleanerLock);
+ }
+ else
+ elog(ERROR, "---> MyWorkerInfo is NULL");
+}
+
+/*
+ * Initialize communication with backends
+ */
+void
+HeapCleanerInit()
+{
+ struct addrinfo *addrs = NULL,
+ *addr, hints;
+ ACCEPT_TYPE_ARG3 alen;
+ int ret;
+ fd_set rset;
+ struct timeval tv;
+ char test_byte;
+ int sel_res;
+ int tries = 0;
+
+#define TESTBYTEVAL ((char) 199)
+
+ /*
+ * Create the UDP socket for sending and receiving statistic messages
+ */
+ hints.ai_flags = AI_PASSIVE;
+ hints.ai_family = AF_UNSPEC;
+ hints.ai_socktype = SOCK_DGRAM;
+ hints.ai_protocol = 0;
+ hints.ai_addrlen = 0;
+ hints.ai_addr = NULL;
+ hints.ai_canonname = NULL;
+ hints.ai_next = NULL;
+ ret = pg_getaddrinfo_all("localhost", NULL, &hints, &addrs);
+ Assert(ret == 0);
+
+ for (addr = addrs; addr; addr = addr->ai_next)
+ {
+#ifdef HAVE_UNIX_SOCKETS
+ /* Ignore AF_UNIX sockets, if any are returned. */
+ if (addr->ai_family == AF_UNIX)
+ continue;
+#endif
+
+ if (++tries > 1)
+ ereport(LOG,
+ (errmsg("trying another address for the statistics collector")));
+
+ if ((HeapCleanerSock = socket(addr->ai_family, SOCK_DGRAM, 0)) == PGINVALID_SOCKET)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not create socket for relation cleaner: %m")));
+ continue;
+ }
+
+ if (bind(HeapCleanerSock, addr->ai_addr, addr->ai_addrlen) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not bind socket for relation cleaner: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ alen = sizeof(HeapCleanerSockAddr);
+ if (getsockname(HeapCleanerSock, (struct sockaddr *) &HeapCleanerSockAddr, &alen) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not get address of socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ if (connect(HeapCleanerSock, (struct sockaddr *) &HeapCleanerSockAddr, alen) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not connect socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+ /*
+ * Try to send and receive a one-byte test message on the socket. This
+ * is to catch situations where the socket can be created but will not
+ * actually pass data (for instance, because kernel packet filtering
+ * rules prevent it).
+ */
+ test_byte = TESTBYTEVAL;
+
+retry1:
+ if (send(HeapCleanerSock, &test_byte, 1, 0) != 1)
+ {
+ if (errno == EINTR)
+ goto retry1; /* if interrupted, just retry */
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not send test message on socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ /*
+ * There could possibly be a little delay before the message can be
+ * received. We arbitrarily allow up to half a second before deciding
+ * it's broken.
+ */
+ for (;;) /* need a loop to handle EINTR */
+ {
+ FD_ZERO(&rset);
+ FD_SET(HeapCleanerSock, &rset);
+
+ tv.tv_sec = 0;
+ tv.tv_usec = 500000;
+ sel_res = select(HeapCleanerSock + 1, &rset, NULL, NULL, &tv);
+ if (sel_res >= 0 || errno != EINTR)
+ break;
+ }
+ if (sel_res < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("select() failed in statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+ if (sel_res == 0 || !FD_ISSET(HeapCleanerSock, &rset))
+ {
+ /*
+ * This is the case we actually think is likely, so take pains to
+ * give a specific message for it.
+ *
+ * errno will not be set meaningfully here, so don't use it.
+ */
+ ereport(LOG,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("test message did not get through on socket for statistics collector")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ test_byte++; /* just make sure variable is changed */
+
+retry2:
+ if (recv(HeapCleanerSock, &test_byte, 1, 0) != 1)
+ {
+ if (errno == EINTR)
+ goto retry2; /* if interrupted, just retry */
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not receive test message on socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ if (test_byte != TESTBYTEVAL) /* strictly paranoia ... */
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("incorrect test message transmission on socket for statistics collector")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ /* If we get here, we have a working socket */
+
+ break;
+ }
+
+ /* Did we find a working address? */
+ if (!addr || HeapCleanerSock == PGINVALID_SOCKET)
+ goto startup_failed;
+
+ /*
+ * Set the socket to non-blocking IO. This ensures that if the collector
+ * falls behind, statistics messages will be discarded; backends won't
+ * block waiting to send messages to the collector.
+ */
+ if (!pg_set_noblock(HeapCleanerSock))
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not set statistics collector socket to nonblocking mode: %m")));
+ goto startup_failed;
+ }
+ pg_freeaddrinfo_all(hints.ai_family, addrs);
+
+ return;
+
+startup_failed:
+ ereport(LOG,
+ (errmsg("disabling heap cleaner for lack of working socket")));
+
+ if (addrs)
+ pg_freeaddrinfo_all(hints.ai_family, addrs);
+
+ if (HeapCleanerSock != PGINVALID_SOCKET)
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+}
+
+/*
+ * Start and initialization logic of a launcher
+ */
+NON_EXEC_STATIC void
+HeapCleanerLauncherMain(int argc, char *argv[])
+{
+ sigjmp_buf local_sigjmp_buf;
+ MemoryContext bgheap_context;
+ int counter;
+
+ am_heapcleaner_launcher = true;
+
+ /*
+ * Identify myself via ps
+ */
+ init_ps_display(pgstat_get_backend_desc(B_BG_HEAPCLNR_LAUNCHER), "", "", "");
+
+ SetProcessingMode(InitProcessing);
+
+ pqsignal(SIGHUP, SIGHUP_Handler); /* set flag to read config file */
+ pqsignal(SIGINT, StatementCancelHandler);
+ pqsignal(SIGTERM, SIGTERM_Handler);
+ pqsignal(SIGQUIT, quickdie); /* hard crash time */
+ InitializeTimeouts();
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIG_IGN);
+ pqsignal(SIGFPE, FloatExceptionHandler);
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ BaseInit();
+
+#ifndef EXEC_BACKEND
+ InitProcess();
+#endif
+
+ InitPostgres(NULL, InvalidOid, NULL, InvalidOid, NULL, false);
+
+ SetProcessingMode(NormalProcessing);
+
+ /*
+ * Create a resource owner to keep track of our resources (currently only
+ * buffer pins).
+ */
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "Background cleaner");
+
+ bgheap_context = AllocSetContextCreate(TopMemoryContext,
+ "Background cleaner",
+ ALLOCSET_DEFAULT_SIZES);
+ MemoryContextSwitchTo(bgheap_context);
+
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Since not using PG_TRY, must reset error stack by hand */
+ error_context_stack = NULL;
+
+ /* Prevent interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Forget any pending QueryCancel or timeout request */
+ disable_all_timeouts(false);
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ LWLockReleaseAll();
+ pgstat_report_wait_end();
+ AbortBufferIO();
+ UnlockBuffers();
+ if (CurrentResourceOwner)
+ /* buffer pins are released here: */
+ ResourceOwnerRelease(CurrentResourceOwner,
+ RESOURCE_RELEASE_BEFORE_LOCKS,
+ false, true);
+ /* we needn't bother with the other ResourceOwnerRelease phases */
+ AtEOXact_Buffers(false);
+ AtEOXact_SMgr();
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
+
+ /*
+ * Now return to normal top-level context and clear ErrorContext for
+ * next time.
+ */
+ MemoryContextSwitchTo(bgheap_context);
+ FlushErrorState();
+
+ /* Flush any leaked data in the top-level context */
+ MemoryContextResetAndDeleteChildren(bgheap_context);
+
+ /*
+ * Make sure pgstat also considers our stat data as gone. Note: we
+ * mustn't use autovac_refresh_stats here.
+ */
+ pgstat_clear_snapshot();
+
+ /* Now we can allow interrupts again */
+ RESUME_INTERRUPTS();
+
+ if (got_SIGTERM)
+ proc_exit(0);
+ /*
+ * Sleep at least 1 second after any error. A write error is likely
+ * to be repeated, and we don't want to be filling the error logs as
+ * fast as we can.
+ */
+ pg_usleep(1000000L);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ /*
+ * Unblock signals (they were blocked when the postmaster forked us)
+ */
+ PG_SETMASK(&UnBlockSig);
+
+ wTabCtl.FillFactor = 0.75;
+ wTabCtl.ElementsMaxNum = WAITING_MESSAGES_MAX_NUM;
+ wTabCtl.ElementSize = sizeof(CleanerMessage);
+ wTabCtl.KeySize = 2 * sizeof(Oid) + sizeof(BlockNumber);
+ wTabCtl.HashFunc = CleanerMessageHashFunc;
+ wTabCtl.CompFunc = isEqualMsgs;
+
+ Assert(heapcleaner_max_workers == 10);
+ wTab = palloc0((heapcleaner_max_workers+1)*sizeof(PSHTAB));
+ for (counter = 0; counter < heapcleaner_max_workers+1; counter++)
+ wTab[counter] = SHASH_Create(wTabCtl);
+ on_shmem_exit(FreeLauncherInfo, 0);
+elog(LOG, "-> Launcher Started!");
+ main_launcher_loop();
+
+ proc_exit(0);
+}
+
+static bool
+backend_store_dirty_block(CleanerMessage *msg)
+{
+ bool found;
+ CleanerMessage *rec;
+
+ /* Initialize collector's hash table */
+ if (dblocks == NULL)
+ {
+ SHTABCTL ctl;
+
+ ctl.ElementsMaxNum = BACKEND_DIRTY_ITEMS_MAX;
+ ctl.FillFactor = 0.75;
+ ctl.ElementSize = sizeof(CleanerMessage);
+ ctl.KeySize = 2 * sizeof(Oid) + sizeof(BlockNumber);
+ ctl.HashFunc = CleanerMessageHashFunc;
+ ctl.CompFunc = isEqualMsgs;
+ dblocks = SHASH_Create(ctl);
+ }
+
+ /* Search for new position or for duplicates */
+ rec = (CleanerMessage *) SHASH_Search(dblocks, (void *) msg, SHASH_ENTER, &found);
+
+ if (!rec)
+ return true;
+
+ if (!found)
+ rec->hits = 0;
+
+ rec->xid = msg->xid;
+ rec->hits++;
+ return false;
+}
+
+/*
+ * Send a package of dirty blocks from a Backend to the Cleaner.
+ * Form the package by passing across the table. During this pass table is
+ * cleaned. Sign of clean row is (dbNode == 0) condition.
+ */
+static void
+backend_send_dirty_blocks(void)
+{
+ int rc;
+ CleanerMessage *rec;
+ CleanerMessage data[BACKEND_DIRTY_ITEMS_MAX];
+ int nitems = 0;
+
+ if (dblocks == NULL)
+ return;
+
+ if (SHASH_Entries(dblocks) == 0)
+ return;
+
+ Assert(SHASH_Entries(dblocks) <= BACKEND_DIRTY_ITEMS_MAX);
+
+ for (SHASH_SeqReset(dblocks);
+ (rec = SHASH_SeqNext(dblocks)) != NULL; )
+ {
+ Assert(rec->hits > 0);
+ Assert(nitems < BACKEND_DIRTY_ITEMS_MAX);
+ data[nitems++] = *rec;
+ }
+ SHASH_Clean(dblocks);
+
+ /* Send data to launcher.
+ * We'll retry after EINTR, but ignore all other failures
+ */
+ do
+ {
+ rc = send(HeapCleanerSock, data, nitems*sizeof(CleanerMessage), 0);
+ } while (rc < 0 && errno == EINTR);
+}
+
+/*
+ * Send a dirty block of relation to the Cleaner.
+ * Before shipping to the Cleaner dirty blocks collects in a simple hash
+ * table. Duplicates are discarded.
+ */
+void
+HeapCleanerSend(Relation relation, BlockNumber blkno)
+{
+ CleanerMessage msg;
+ MemoryContext oldMemCxt;
+
+ if (BGHeapMemCxt == NULL)
+ BGHeapMemCxt = AllocSetContextCreate(BGHeapMemCxt, "bgheap",
+ ALLOCSET_DEFAULT_SIZES);
+
+ if (RecoveryInProgress())
+ return;
+
+ if (HeapCleanerSock == PGINVALID_SOCKET)
+ return;
+
+ /*
+ * ToDo: Cleaning of system relations make instability in regression tests
+ * output order
+ */
+ if (IsSystemRelation(relation))
+ return;
+
+ /*
+ * Check the relation type.
+ * Not all relations can be cleaned now (similarly vacuum).
+ */
+ if (relation->rd_rel->relkind != RELKIND_RELATION &&
+ relation->rd_rel->relkind != RELKIND_MATVIEW &&
+ relation->rd_rel->relkind != RELKIND_TOASTVALUE &&
+ relation->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
+ {
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- cannot clean non-tables or special system tables",
+ RelationGetRelationName(relation))));
+ return;
+ }
+
+ if (RELATION_IS_OTHER_TEMP(relation))
+ return;
+
+ if (relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ return;
+
+ oldMemCxt = MemoryContextSwitchTo(BGHeapMemCxt);
+
+ msg.dbNode = relation->rd_node.dbNode;
+ msg.relid = RelationGetRelid(relation);
+ msg.blkno = blkno;
+ msg.xid = GetCurrentTransactionIdIfAny();
+
+ if (backend_store_dirty_block(&msg))
+ backend_send_dirty_blocks();
+
+ MemoryContextSwitchTo(oldMemCxt);
+}
+
+/*
+ * Start and initialization logic of a worker
+ */
+NON_EXEC_STATIC void
+HeapCleanerWorkerMain(int argc, char *argv[])
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ am_heapcleaner_worker = true;
+
+ /* Identify myself via ps */
+ init_ps_display(pgstat_get_backend_desc(B_BG_HEAPCLNR_WORKER), "", "", "");
+
+ SetProcessingMode(InitProcessing);
+
+ pqsignal(SIGHUP, SIGHUP_Handler);
+ pqsignal(SIGINT, StatementCancelHandler);
+ pqsignal(SIGTERM, SIGTERM_Handler);
+ pqsignal(SIGQUIT, quickdie);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
+
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIGUSR2_Handler); /* Signal: Buffer contains a message */
+ pqsignal(SIGFPE, FloatExceptionHandler);
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Early initialization */
+ BaseInit();
+
+#ifndef EXEC_BACKEND
+ InitProcess();
+#endif
+
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Prevents interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ proc_exit(0);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ PG_SETMASK(&UnBlockSig);
+
+ /* Initialization steps */
+ LWLockAcquire(HeapCleanerLock, LW_EXCLUSIVE);
+ if (HeapCleanerShmem->startingWorker != NULL)
+ {
+ MyWorkerInfo = HeapCleanerShmem->startingWorker;
+ MyWorkerInfo->pid = getpid();
+ HeapCleanerShmem->startingWorker = NULL;
+
+ /* insert into the running list */
+ dlist_push_head(&HeapCleanerShmem->runningWorkers, &MyWorkerInfo->links);
+
+ on_shmem_exit(FreeWorkerInfo, 0);
+ }
+ else
+ elog(ERROR, "No Starting worker!");
+ LWLockRelease(HeapCleanerLock);
+
+ if (OidIsValid(MyWorkerInfo->dbOid))
+ {
+ char dbname[NAMEDATALEN];
+ SHTABCTL pr_ctl;
+
+ pgstat_report_heapcleaner(MyWorkerInfo->dbOid);
+ InitPostgres(NULL, MyWorkerInfo->dbOid, NULL, InvalidOid, dbname, false);
+ SetProcessingMode(NormalProcessing);
+ set_ps_display(dbname, false);
+
+ pr_ctl.ElementsMaxNum = 1000;
+ pr_ctl.FillFactor = 0.5;
+ pr_ctl.HashFunc = DefaultHashValueFunc;
+ pr_ctl.CompFunc = DefaultCompareFunc;
+ pr_ctl.ElementSize = sizeof(DirtyRelation);
+ pr_ctl.KeySize = sizeof(int32);
+
+ PrivateRelationsTable = SHASH_Create(pr_ctl);
+
+ /* Add tracking info to pgstat */
+ pgstat_progress_start_command(PROGRESS_COMMAND_CLEANER, MyWorkerInfo->dbOid);
+
+ main_worker_loop();
+ }
+ else
+ elog(ERROR, "dbid %d not valid!", MyWorkerInfo->dbOid);
+
+ proc_exit(0);
+}
+
+/*
+ * Compare messages
+ */
+static bool
+isEqualMsgs(void *arg1, void *arg2)
+{
+ CleanerMessage *msg1 = arg1;
+ CleanerMessage *msg2 = arg2;
+
+ Assert(arg1 != NULL);
+ Assert(arg2 != NULL);
+
+ if (msg1->blkno != msg2->blkno)
+ return false;
+ if (msg1->relid != msg2->relid)
+ return false;
+ if (msg1->dbNode != msg2->dbNode)
+ return false;
+
+ return true;
+}
+
+/*
+ * IsHeapCleaner functions
+ * Return whether this is either a launcher heap cleaner process or
+ * a worker process.
+ */
+bool
+IsHeapCleanerLauncherProcess(void)
+{
+ return am_heapcleaner_launcher;
+}
+
+bool
+IsHeapCleanerWorkerProcess(void)
+{
+ return am_heapcleaner_worker;
+}
+
+/*
+ * Send signal to Postmaster for launch new worker instance.
+ * Eject one node from freeWorkers list and assign to startingWorker
+ * HeapCleanerLock must be exclusive-locked
+ */
+static void
+launch_worker(Oid dbNode)
+{
+ WorkerInfo worker;
+ dlist_node *wptr;
+
+ if (dlist_is_empty(&HeapCleanerShmem->freeWorkers))
+ elog(ERROR, "NO a free slot for background cleaner worker!");
+
+ wptr = dlist_pop_head_node(&HeapCleanerShmem->freeWorkers);
+ worker = dlist_container(WorkerInfoData, links, wptr);
+ worker->dbOid = dbNode;
+ worker->launchtime = GetCurrentTimestamp();
+ worker->nitems = 0;
+ HeapCleanerShmem->startingWorker = worker;
+
+ SendPostmasterSignal(PMSIGNAL_START_HEAPCLNR_WORKER);
+}
+
+/*
+ * Return worker that initialized for dbNode database or NULL
+ * Caller need to acquire share lock on HeapCleanerLock
+ */
+static WorkerInfo
+look_for_worker(Oid dbNode)
+{
+ dlist_node *node;
+ WorkerInfo worker = NULL;
+
+ if (!dlist_is_empty(&HeapCleanerShmem->runningWorkers))
+ for (node = dlist_head_node(&HeapCleanerShmem->runningWorkers);
+ ;
+ node = dlist_next_node(&HeapCleanerShmem->runningWorkers, node))
+ {
+ if (((WorkerInfo)node)->dbOid == dbNode)
+ {
+ worker = (WorkerInfo) node;
+ break;
+ }
+ if (!dlist_has_next(&HeapCleanerShmem->runningWorkers, node))
+ break;
+ }
+
+ return worker;
+}
+
+/*
+ * Entry point of a launcher behavior logic
+ */
+static void
+main_launcher_loop()
+{
+ bool found;
+ long timeout = -1L;
+
+ Assert(HeapCleanerSock != PGINVALID_SOCKET);
+
+ /*
+ * Do work until not catch a SIGTERM signal and all blocks
+ * not shipped to workers.
+ * timeout <= 0 tells us that the launcher does not have any work right now,
+ * and it will sleep until the socket message or the system signal.
+ */
+ while (!got_SIGTERM || (timeout > 0))
+ {
+ int rc;
+ int len;
+ CleanerMessage *msg;
+ CleanerMessage table[BACKEND_DIRTY_ITEMS_MAX];
+ WorkerInfo startingWorker;
+ WorkerInfo worker;
+ dlist_node *node;
+
+ timeout = -1L;
+
+ /* Process system signals */
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ ResetLatch(MyLatch);
+
+ /* At First, receive a message from backend */
+ len = recv(HeapCleanerSock, table, BACKEND_DIRTY_ITEMS_MAX*sizeof(CleanerMessage), 0);
+
+ LWLockAcquire(HeapCleanerLock, LW_EXCLUSIVE);
+ startingWorker = HeapCleanerShmem->startingWorker;
+
+ /*
+ * Message was received from socket.
+ */
+ if (len > 0)
+ {
+ CleanerMessage *mptr;
+
+ if (len%sizeof(CleanerMessage) != 0)
+ elog(ERROR, "INCORRECT Message size!");
+
+ /*
+ * Push all messages received from a backend to general
+ * hash table with waiting messages.
+ */
+ for (mptr = table; ((char *)mptr-(char *)table) < len; mptr++)
+ {
+ worker = look_for_worker(mptr->dbNode);
+ if (worker)
+ msg = (CleanerMessage *) SHASH_Search(wTab[worker->id], mptr, SHASH_ENTER, &found);
+ else
+ msg = (CleanerMessage *) SHASH_Search(wTab[heapcleaner_max_workers], mptr, SHASH_ENTER, &found);
+
+ if (msg == NULL)
+ /*
+ * Hash table is FULL and we can't insert a message
+ */
+ stat_missed_blocks++;
+ else if (found)
+ {
+ msg->hits++;
+ msg->xid = (msg->xid > mptr->xid) ? msg->xid : mptr->xid;
+ }
+ else
+ memcpy(msg, mptr, sizeof(CleanerMessage));
+ }
+ }
+
+ /*
+ * Main waiting list parsing.
+ */
+ if (SHASH_Entries(wTab[heapcleaner_max_workers]) > 0)
+ {
+ bool startWorker = (startingWorker != NULL);
+
+ /*
+ * Pass across general waiting list.
+ * Try to send tasks to active workers.
+ */
+ for (SHASH_SeqReset(wTab[heapcleaner_max_workers]);
+ (msg = (CleanerMessage *) SHASH_SeqNext(wTab[heapcleaner_max_workers])) != NULL; )
+ {
+ worker = look_for_worker(msg->dbNode);
+
+ if (worker)
+ {
+ CleanerMessage *temp_msg = (CleanerMessage *) SHASH_Search(wTab[worker->id], (void *) msg, SHASH_ENTER, &found);
+ if (temp_msg != NULL)
+ if (found)
+ {
+ temp_msg->hits += msg->hits;
+ temp_msg->xid = (msg->xid > temp_msg->xid) ? msg->xid : temp_msg->xid;
+ }
+ else
+ memcpy(temp_msg, msg, sizeof(CleanerMessage));
+ else
+ stat_missed_blocks++;
+
+ /*
+ * Message gone to worker. delete from main waiting list.
+ */
+ temp_msg = (CleanerMessage *) SHASH_Search(wTab[heapcleaner_max_workers], (void *) msg, SHASH_REMOVE, NULL);
+ Assert(temp_msg != NULL);
+
+ continue;
+ }
+ else if (!startWorker)
+ {
+ /* Start new worker */
+ launch_worker(msg->dbNode);
+ startWorker = true;
+ }
+ }
+ }
+
+ /*
+ * Check: if we can't process all incoming messages, we need too small
+ * timeout, check latches and go to next iteration.
+ */
+ if (SHASH_Entries(wTab[heapcleaner_max_workers]) > 0)
+ timeout = 1L;
+
+ /*
+ * See waiting lists of active workers and try to send messages.
+ */
+ if (!dlist_is_empty(&HeapCleanerShmem->runningWorkers))
+ {
+ for (node = dlist_head_node(&HeapCleanerShmem->runningWorkers);
+ ;
+ node = dlist_next_node(&HeapCleanerShmem->runningWorkers, node))
+ {
+ worker = (WorkerInfo) node;
+
+ if (SHASH_Entries(wTab[worker->id]) == 0)
+ {
+ /* Check worker idle time */
+ if (TimestampDifferenceExceeds(worker->launchtime, GetCurrentTimestamp(), WORKER_IDLE_TIME_DURATION_MAX))
+ {
+ LWLockAcquire(&worker->WorkItemLock, LW_EXCLUSIVE);
+
+ /*
+ * Check: may be worker has tasks but is too lazy
+ */
+ if (worker->nitems == 0)
+ {
+ /* Shutdown the idle worker */
+ kill(worker->pid, SIGTERM);
+ dlist_delete(&worker->links);
+ dlist_push_head(&HeapCleanerShmem->freeWorkers, &worker->links);
+
+ if (!dlist_is_empty(&HeapCleanerShmem->runningWorkers))
+ node = dlist_head_node(&HeapCleanerShmem->runningWorkers);
+ else
+ {
+ LWLockRelease(&worker->WorkItemLock);
+ break;
+ }
+ }
+ LWLockRelease(&worker->WorkItemLock);
+ }
+ else if (!dlist_has_next(&HeapCleanerShmem->runningWorkers, node))
+ break;
+ else
+ continue;
+ }
+
+ /* Put list of potentially dirty blocks to the worker shared buffer */
+ LWLockAcquire(&worker->WorkItemLock, LW_EXCLUSIVE);
+
+ SHASH_SeqReset(wTab[worker->id]);
+ while (((msg = (CleanerMessage *) SHASH_SeqNext(wTab[worker->id])) != NULL) &&
+ (worker->nitems < WORKER_TASK_ITEMS_MAX))
+ {
+ void *temp_msg;
+
+ memcpy(&worker->buffer[worker->nitems++], msg, sizeof(CleanerMessage));
+ temp_msg = SHASH_Search(wTab[worker->id], (void *) msg, SHASH_REMOVE, NULL);
+ Assert(temp_msg != NULL);
+ Assert(worker->buffer[worker->nitems-1].hits > 0);
+ }
+
+ worker->launchtime = GetCurrentTimestamp();
+ LWLockRelease(&worker->WorkItemLock);
+
+ /* Worker, you have a task! */
+ kill(worker->pid, SIGUSR2);
+
+ /*
+ * If launcher has any tasks, It check signals quickly
+ * and to work on further.
+ */
+ if (SHASH_Entries(wTab[worker->id]) > 0)
+ timeout = 5L;
+
+ if (!dlist_has_next(&HeapCleanerShmem->runningWorkers, node))
+ break;
+ }
+
+ /*
+ * Launcher have'nt any immediate tasks and
+ * need to manage idle workers only
+ */
+ if (timeout < 0)
+ /* We only need to wait idle workers */
+ timeout = 100L;
+ }
+ LWLockRelease(HeapCleanerLock);
+
+ if (!got_SIGTERM)
+ {
+ int wakeEvents = WL_LATCH_SET | WL_POSTMASTER_DEATH | WL_SOCKET_READABLE;
+
+ if (timeout > 0)
+ wakeEvents |= WL_TIMEOUT;
+ /* Wait data or signals */
+ rc = WaitLatchOrSocket(MyLatch,
+ wakeEvents,
+ HeapCleanerSock, timeout,
+ WAIT_EVENT_BGHEAP_MAIN);
+ }
+
+ /* Emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ {
+ elog(LOG, "Heap Launcher exit with 1");
+ proc_exit(1);
+ }
+ }
+
+ elog(LOG, "Heap Launcher exit with 0");
+ proc_exit(0);
+}
+
+/*
+ * Entry point of a worker behavior logic
+ */
+static void
+main_worker_loop(void)
+{
+ CleanerMessage task_item[WORKER_TASK_ITEMS_MAX];
+ DirtyRelation *dirty_relation[WORKER_RELATIONS_MAX_NUM];
+ PSHTAB FreeDirtyBlocksList;
+ long timeout = -1L;
+ int dirty_relations_num = 0;
+ int task_items_num = 0;
+ SHTABCTL shctl;
+
+ shctl.FillFactor = 0.75;
+ shctl.ElementsMaxNum = WORKER_DIRTYBLOCKS_MAX_NUM;
+ shctl.ElementSize = sizeof(WorkerTask);
+ shctl.KeySize = sizeof(BlockNumber);
+ shctl.HashFunc = DefaultHashValueFunc;
+ shctl.CompFunc = DefaultCompareFunc;
+
+ FreeDirtyBlocksList = SHASH_Create(shctl);
+
+ while (!got_SIGTERM || (timeout > 0))
+ {
+ int rc;
+
+ timeout = -1L;
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ if (got_SIGUSR2)
+ /* It is needed only for wakeup worker */
+ got_SIGUSR2 = false;
+
+ /*
+ * Move task items from shared buffer to local and release it for
+ * new data.
+ * It is introduced in accordance with the idea that shared memory
+ * buffer is smaller than internal process buffer.
+ */
+ LWLockAcquire(&MyWorkerInfo->WorkItemLock, LW_SHARED);
+ if (MyWorkerInfo->nitems > 0)
+ {
+ task_items_num = MyWorkerInfo->nitems;
+ memcpy(task_item, MyWorkerInfo->buffer, task_items_num*sizeof(CleanerMessage));
+ MyWorkerInfo->nitems = 0;
+ }
+ LWLockRelease(&MyWorkerInfo->WorkItemLock);
+
+ /*
+ * If launcher receive some task items it need to distribute between
+ * workers and waiting lists.
+ */
+ if (task_items_num > 0)
+ {
+ DirtyRelation *hashent;
+ bool found;
+ int i;
+
+ pgstat_progress_update_param(PROGRESS_CLEANER_MISSED_BLOCKS, stat_missed_blocks);
+
+ /* Pass across items and sort by relation */
+ for (i = 0; i < task_items_num; i++)
+ {
+ WorkerTask *item;
+
+ hashent = SHASH_Search(PrivateRelationsTable,
+ (void *) &(task_item[i].relid), HASH_FIND, NULL);
+
+ if (hashent == NULL)
+ {
+ elog(LOG, "Add new relation: %d", task_item[i].relid);
+ if (dirty_relations_num == WORKER_RELATIONS_MAX_NUM)
+ {
+ int j;
+ int min = SHASH_Entries(dirty_relation[0]->items);
+ int pos = 0;
+
+ /*
+ * All slots for dirty relations are busy.
+ * Search for slot with smaller number of dirty blocks
+ * and replace it.
+ */
+ elog(LOG, "All dirty relation slots are busy at worker process. Start min=%d", min);
+ for (j = 1; j < dirty_relations_num; j++)
+ {
+ int nitems = SHASH_Entries(dirty_relation[j]->items);
+ if (nitems < min)
+ {
+ min = nitems;
+ pos = j;
+ }
+ }
+
+ /* Save stats about not cleaned blocks */
+ stat_missed_blocks += SHASH_Entries(dirty_relation[pos]->items);
+
+ elog(LOG, "Remove slot %d with id=%d nitems=%lu. Set new relation id=%d",pos, dirty_relation[pos]->relid, SHASH_Entries(dirty_relation[pos]->items), task_item[i].relid);
+ hashent = SHASH_Search(PrivateRelationsTable,
+ (void *) &(dirty_relation[pos]->relid), HASH_REMOVE, &found);
+
+ Assert(found);
+ Assert(hashent != NULL);
+ SHASH_Destroy(hashent->items);
+
+ /*
+ * Fill empty position by data from last position
+ */
+ dirty_relation[pos] = dirty_relation[--dirty_relations_num];
+ }
+
+ /* Insert new relid to hash table */
+ hashent = SHASH_Search(PrivateRelationsTable, (void *) &(task_item[i].relid), HASH_ENTER, &found);
+ Assert(!found);
+ Assert(hashent != NULL);
+
+ /*
+ * At a new entry we create and init list of 'dirty' blocks
+ */
+ hashent->items = SHASH_Create(shctl);
+ dirty_relation[dirty_relations_num++] = hashent;
+ pgstat_progress_update_param(PROGRESS_CLEANER_RELATIONS, SHASH_Entries(PrivateRelationsTable));
+ }
+
+ /*
+ * Relation entry found or create.
+ * Now, add an item to a waiting list.
+ */
+ item = (WorkerTask *) SHASH_Search(hashent->items, (void *) &(task_item[i].blkno), HASH_ENTER, &found);
+ if (item == NULL)
+ stat_missed_blocks++;
+ else
+ {
+ if (!found)
+ {
+ item->hits = 0;
+ item->lastXid = InvalidTransactionId;
+ }
+
+ item->hits += task_item[i].hits;
+ Assert(item->hits > 0);
+ item->lastXid = (item->lastXid > task_item[i].xid) ? item->lastXid : task_item[i].xid;
+ }
+ }
+ task_items_num = 0;
+ }
+
+ PG_TRY();
+ {
+ int relcounter;
+ int64 stat_tot_wait_queue_len = 0;
+
+ /* Pass along dirty relations and try to clean it */
+ for (relcounter = 0; relcounter < dirty_relations_num; relcounter++)
+ {
+ if (SHASH_Entries(FreeDirtyBlocksList) > 0)
+ SHASH_Clean(FreeDirtyBlocksList);
+
+ FreeDirtyBlocksList = cleanup_relations(dirty_relation[relcounter], FreeDirtyBlocksList, got_SIGTERM);
+
+ /*
+ * Some blocks from the list may blocked by a backend.
+ * Deferred its for next cleanup attempt.
+ */
+ if ((stat_tot_wait_queue_len += SHASH_Entries(dirty_relation[relcounter]->items)) > 0)
+ timeout = 10L;
+ }
+
+ pgstat_progress_update_param(PROGRESS_CLEANER_TOTAL_QUEUE_LENGTH, stat_tot_wait_queue_len);
+ QueryCancelPending = false;
+ }
+ PG_CATCH();
+ {
+ HOLD_INTERRUPTS();
+ EmitErrorReport();
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ RESUME_INTERRUPTS();
+ }
+ PG_END_TRY();
+
+ if (!got_SIGTERM)
+ {
+ int wakeEvents = WL_LATCH_SET | WL_POSTMASTER_DEATH;
+ if (timeout > 0)
+ wakeEvents |= WL_TIMEOUT;
+ rc = WaitLatch(MyLatch, wakeEvents, timeout, WAIT_EVENT_BGHEAP_MAIN);
+ }
+
+ ResetLatch(MyLatch);
+
+ /* Emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ pgstat_progress_end_command();
+ proc_exit(0);
+}
+
+/*
+ * SIGHUP_Handler
+ */
+static void
+SIGHUP_Handler(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * SIGTERM_Handler
+ */
+static void
+SIGTERM_Handler(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGTERM = true;
+
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * SIGUSR2_Handler
+ * Wake up a worker to read some messages from launcher
+ */
+static void
+SIGUSR2_Handler(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGUSR2 = true;
+
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Main entry point for background heap cleaner (launcher) process, to be called from the
+ * postmaster.
+ */
+int
+StartHeapCleanerLauncher(void)
+{
+ pid_t HeapCleanerLauncherPID;
+
+#ifdef EXEC_BACKEND
+ switch ((HeapCleanerLauncherPID = bghclauncher_forkexec()))
+#else
+ switch ((HeapCleanerLauncherPID = fork_process()))
+#endif
+ {
+ case -1:
+ ereport(LOG,
+ (errmsg("could not fork background heap cleaner (launcher) process: %m")));
+ return 0;
+
+#ifndef EXEC_BACKEND
+ case 0:
+ /* in postmaster child ... */
+ InitPostmasterChild();
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ HeapCleanerLauncherMain(0, NULL);
+ break;
+#endif
+ default:
+ return (int) HeapCleanerLauncherPID;
+ }
+
+ /* shouldn't get here */
+ return 0;
+}
+
+/*
+ * Main entry point for background heap cleaner (worker) process, to be called from the
+ * postmaster.
+ */
+int
+StartHeapCleanerWorker(void)
+{
+ pid_t worker_pid;
+
+#ifdef EXEC_BACKEND
+ switch ((worker_pid = hcworker_forkexec()))
+#else
+ switch ((worker_pid = fork_process()))
+#endif
+ {
+ case -1:
+ ereport(LOG,
+ (errmsg("could not fork background heap cleaner (worker) process: %m")));
+ return 0;
+
+#ifndef EXEC_BACKEND
+ case 0:
+ /* in postmaster child ... */
+ InitPostmasterChild();
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ HeapCleanerWorkerMain(0, NULL);
+ break;
+#endif
+ default:
+ return (int) worker_pid;
+ }
+
+ /* shouldn't get here */
+ return 0;
+}
+
+/*
+ * HeapCleanerShmemInit
+ * Allocate and initialize heapcleaner-related shared memory
+ */
+void
+HeapCleanerShmemInit(void)
+{
+ bool found;
+
+ HeapCleanerShmem = (HeapCleanerShmemStruct *)
+ ShmemInitStruct("HeapCleaner Data",
+ HeapCleanerShmemSize(),
+ &found);
+
+ if (!IsUnderPostmaster)
+ {
+ WorkerInfo worker;
+ int i;
+
+ Assert(!found);
+
+ dlist_init(&HeapCleanerShmem->freeWorkers);
+ dlist_init(&HeapCleanerShmem->runningWorkers);
+
+ worker = (WorkerInfo) ((char *) HeapCleanerShmem +
+ MAXALIGN(sizeof(HeapCleanerShmemStruct)));
+
+ /* initialize the WorkerInfo free list */
+ for (i = 0; i < heapcleaner_max_workers; i++)
+ {
+ worker[i].id = i;
+ LWLockInitialize(&worker[i].WorkItemLock, TrancheId);
+ dlist_push_head(&HeapCleanerShmem->freeWorkers,
+ &worker[i].links);
+ }
+ }
+ else
+ Assert(found);
+ HeapCleanerShmem->startingWorker = NULL;
+
+ TrancheId = LWLockNewTrancheId();
+ LWLockRegisterTranche(TrancheId, "heapcleaner");
+}
+
+/*
+ * HeapCleanerShmemSiz
+ * Compute space needed for heap cleaner-related shared memory
+ */
+Size
+HeapCleanerShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of WorkerInfoData.
+ */
+ size = sizeof(HeapCleanerShmemStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(heapcleaner_max_workers,
+ sizeof(WorkerInfoData)));
+ return size;
+}
+
+static uint64
+CleanerMessageHashFunc(void *key, uint64 size, uint64 base)
+{
+ uint64 sum = 0;
+ CleanerMessage *msg = (CleanerMessage *) key;
+
+ Assert(key != NULL);
+ Assert(size == 2 * sizeof(Oid) + sizeof(BlockNumber));
+ Assert(base > 0);
+
+ sum += DefaultHashValueFunc(&msg->dbNode, sizeof(Oid), base);
+ sum += DefaultHashValueFunc(&msg->relid, sizeof(Oid), base);
+ sum += DefaultHashValueFunc(&msg->blkno, sizeof(BlockNumber), base);
+
+ return sum%base;
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index bbe73618c7..f992307e37 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -48,6 +48,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/fork_process.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
@@ -1394,6 +1395,20 @@ pgstat_report_autovac(Oid dboid)
pgstat_send(&msg, sizeof(msg));
}
+void
+pgstat_report_heapcleaner(Oid dboid)
+{
+ PgStat_MsgAutovacStart msg;
+
+ if (pgStatSock == PGINVALID_SOCKET)
+ return;
+
+ pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_BGHEAP_START);
+ msg.m_databaseid = dboid;
+ msg.m_start_time = GetCurrentTimestamp();
+
+ pgstat_send(&msg, sizeof(msg));
+}
/* ---------
* pgstat_report_vacuum() -
@@ -2833,6 +2848,14 @@ pgstat_bestart(void)
/* Autovacuum Worker */
beentry->st_backendType = B_AUTOVAC_WORKER;
}
+ else if (IsHeapCleanerLauncherProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_LAUNCHER;
+ }
+ else if (IsHeapCleanerWorkerProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_WORKER;
+ }
else if (am_walsender)
{
/* Wal sender */
@@ -3483,6 +3506,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_AUTOVACUUM_MAIN:
event_name = "AutoVacuumMain";
break;
+ case WAIT_EVENT_BGHEAP_MAIN:
+ event_name = "BgHeapMain";
+ break;
case WAIT_EVENT_BGWRITER_HIBERNATE:
event_name = "BgWriterHibernate";
break;
@@ -4111,6 +4137,12 @@ pgstat_get_backend_desc(BackendType backendType)
case B_AUTOVAC_WORKER:
backendDesc = "autovacuum worker";
break;
+ case B_BG_HEAPCLNR_LAUNCHER:
+ backendDesc = "heap cleaner launcher";
+ break;
+ case B_BG_HEAPCLNR_WORKER:
+ backendDesc = "heap cleaner worker";
+ break;
case B_BACKEND:
backendDesc = "client backend";
break;
@@ -4422,6 +4454,10 @@ PgstatCollectorMain(int argc, char *argv[])
pgstat_recv_autovac((PgStat_MsgAutovacStart *) &msg, len);
break;
+ case PGSTAT_MTYPE_BGHEAP_START:
+// pgstat_recv_autovac((PgStat_MsgHeapCleanerStart *) &msg, len);
+ break;
+
case PGSTAT_MTYPE_VACUUM:
pgstat_recv_vacuum((PgStat_MsgVacuum *) &msg, len);
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b33cd..d14c1b557d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -109,6 +109,7 @@
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
@@ -145,9 +146,10 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_BGHEAP 0x0010 /* autovacuum worker process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
-#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
+#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP)
/*
* List of active backends (or child processes anyway; we don't actually
@@ -252,6 +254,7 @@ static pid_t StartupPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
+ HeapClnrPID = 0,
PgArchPID = 0,
PgStatPID = 0,
SysLoggerPID = 0;
@@ -355,6 +358,7 @@ bool redirection_done = false; /* stderr redirected for syslogger? */
/* received START_AUTOVAC_LAUNCHER signal */
static volatile sig_atomic_t start_autovac_launcher = false;
+static volatile sig_atomic_t start_heapclnr_launcher = false;
/* the launcher needs to be signalled to communicate some condition */
static volatile bool avlauncher_needs_signal = false;
@@ -432,6 +436,7 @@ static void maybe_start_bgworkers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
+static void StartBgHeapWorker(void);
static void MaybeStartWalReceiver(void);
static void InitPostmasterDeathWatchHandle(void);
@@ -1310,6 +1315,7 @@ PostmasterMain(int argc, char *argv[])
*/
autovac_init();
+ HeapCleanerInit();
/*
* Load configuration files for client authentication.
*/
@@ -1757,6 +1763,14 @@ ServerLoop(void)
start_autovac_launcher = false; /* signal processed */
}
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && start_heapclnr_launcher &&
+ pmState == PM_RUN)
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ if (HeapClnrPID != 0)
+ start_heapclnr_launcher = false; /* signal processed */
+ }
+
/* If we have lost the stats collector, try to start a new one */
if (PgStatPID == 0 &&
(pmState == PM_RUN || pmState == PM_HOT_STANDBY))
@@ -1774,6 +1788,12 @@ ServerLoop(void)
kill(AutoVacPID, SIGUSR2);
}
+ /* If we have lost the background heap cleaner, try to start a new one. */
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && pmState == PM_RUN )
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ }
+
/* If we need to start a WAL receiver, try to do that now */
if (WalReceiverRequested)
MaybeStartWalReceiver();
@@ -2541,6 +2561,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGHUP);
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGHUP);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGHUP);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGHUP);
if (SysLoggerPID != 0)
@@ -2627,10 +2649,12 @@ pmdie(SIGNAL_ARGS)
/* autovac workers are told to shut down immediately */
/* and bgworkers too; does this need tweaking? */
SignalSomeChildren(SIGTERM,
- BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the bgwriter too */
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
@@ -2687,7 +2711,7 @@ pmdie(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGTERM);
if (pmState == PM_RECOVERY)
{
- SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER);
+ SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/*
* Only startup, bgwriter, walreceiver, possibly bgworkers,
@@ -2708,10 +2732,12 @@ pmdie(SIGNAL_ARGS)
/* shut down all backends and workers */
SignalSomeChildren(SIGTERM,
BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC |
- BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the walwriter too */
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGTERM);
@@ -2875,6 +2901,8 @@ reaper(SIGNAL_ARGS)
*/
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
+ if (!IsBinaryUpgrade && (HeapClnrPID == 0))
+ HeapClnrPID = StartHeapCleanerLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
PgArchPID = pgarch_start();
if (PgStatPID == 0)
@@ -3009,6 +3037,14 @@ reaper(SIGNAL_ARGS)
_("autovacuum launcher process"));
continue;
}
+ if (pid == HeapClnrPID)
+ {
+ HeapClnrPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("heap cleaner launcher process"));
+ continue;
+ }
/*
* Was it the archiver? If so, just try to start a new one; no need
@@ -3475,6 +3511,17 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ if (pid == HeapClnrPID)
+ HeapClnrPID = 0;
+ else if (HeapClnrPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ "SIGQUIT",
+ (int) HeapClnrPID)));
+ signal_child(HeapClnrPID, SIGQUIT);
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3661,6 +3708,7 @@ PostmasterStateMachine(void)
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
WalWriterPID == 0 &&
+ HeapClnrPID == 0 &&
AutoVacPID == 0)
{
if (Shutdown >= ImmediateShutdown || FatalError)
@@ -3759,6 +3807,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(HeapClnrPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -3948,6 +3997,8 @@ TerminateChildren(int signal)
signal_child(WalReceiverPID, signal);
if (AutoVacPID != 0)
signal_child(AutoVacPID, signal);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, signal);
if (PgArchPID != 0)
signal_child(PgArchPID, signal);
if (PgStatPID != 0)
@@ -4781,6 +4832,7 @@ SubPostmasterMain(int argc, char *argv[])
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
+ strcmp(argv[1], "--forkbgheap") == 0 ||
strcmp(argv[1], "--forkboot") == 0 ||
strncmp(argv[1], "--forkbgworker=", 15) == 0)
PGSharedMemoryReAttach();
@@ -4792,6 +4844,8 @@ SubPostmasterMain(int argc, char *argv[])
AutovacuumLauncherIAm();
if (strcmp(argv[1], "--forkavworker") == 0)
AutovacuumWorkerIAm();
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ BgHeapCleanerIAm();
/*
* Start our win32 signal implementation. This has to be done after we
@@ -4921,6 +4975,19 @@ SubPostmasterMain(int argc, char *argv[])
AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores(false, 0);
+
+ BgHeapMain(argc, argv); /* does not return */
+ }
if (strncmp(argv[1], "--forkbgworker=", 15) == 0)
{
int shmem_slot;
@@ -5122,6 +5189,12 @@ sigusr1_handler(SIGNAL_ARGS)
start_autovac_launcher = true;
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_LAUNCHER) &&
+ Shutdown == NoShutdown)
+ {
+ start_heapclnr_launcher = true;
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_AUTOVAC_WORKER) &&
Shutdown == NoShutdown)
{
@@ -5129,6 +5202,13 @@ sigusr1_handler(SIGNAL_ARGS)
StartAutovacuumWorker();
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_WORKER) &&
+ Shutdown == NoShutdown)
+ {
+ /* The autovacuum launcher wants us to start a worker process. */
+ StartBgHeapWorker();
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_WALRECEIVER))
{
/* Startup Process wants us to start the walreceiver process. */
@@ -5470,6 +5550,73 @@ StartAutovacuumWorker(void)
}
}
+static void
+StartBgHeapWorker(void)
+{
+ Backend *bn;
+
+ /*
+ * If not in condition to run a process, don't try, but handle it like a
+ * fork failure. This does not normally happen, since the signal is only
+ * supposed to be sent by autovacuum launcher when it's OK to do it, but
+ * we have to check to avoid race-condition problems during DB state
+ * changes.
+ */
+ if (canAcceptConnections() == CAC_OK)
+ {
+ /*
+ * Compute the cancel key that will be assigned to this session. We
+ * probably don't need cancel keys for autovac workers, but we'd
+ * better have something random in the field to prevent unfriendly
+ * people from sending cancels to them.
+ */
+ if (!RandomCancelKey(&MyCancelKey))
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random cancel key")));
+ return;
+ }
+
+ bn = (Backend *) malloc(sizeof(Backend));
+ if (bn)
+ {
+ bn->cancel_key = MyCancelKey;
+
+ /* Autovac workers are not dead_end and need a child slot */
+ bn->dead_end = false;
+ bn->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+ bn->bgworker_notify = false;
+
+ bn->pid = StartHeapCleanerWorker();
+
+ if (bn->pid > 0)
+ {
+ bn->bkend_type = BACKEND_TYPE_BGHEAP;
+ dlist_push_head(&BackendList, &bn->elem);
+#ifdef EXEC_BACKEND
+ ShmemBackendArrayAdd(bn);
+#endif
+ /* all OK */
+ return;
+ }
+
+ /*
+ * fork failed, fall through to report -- actual error message was
+ * logged by StartAutoVacWorker
+ */
+ (void) ReleasePostmasterChildSlot(bn->child_slot);
+ free(bn);
+ }
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+ }
+
+ elog(ERROR, "Some BGHEAP Worker start error!");
+}
+
/*
* MaybeStartWalReceiver
* Start the WAL receiver process, if not running and our state allows.
@@ -5534,7 +5681,8 @@ CreateOptsFile(int argc, char *argv[], char *fullprogname)
int
MaxLivePostmasterChildren(void)
{
- return 2 * (MaxConnections + autovacuum_max_workers + 1 +
+ return 2 * (MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 +
max_worker_processes);
}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 01eabe5706..175902e0d3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -449,6 +449,7 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
+ bool onlyInMemory,
bool *foundPtr);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
static void AtProcExit_Buffers(int code, Datum arg);
@@ -717,6 +718,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
isExtend = (blockNum == P_NEW);
+ Assert(!(isExtend && mode == RBM_NORMAL_NO_READ));
+
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
smgr->smgr_rnode.node.spcNode,
smgr->smgr_rnode.node.dbNode,
@@ -743,9 +746,11 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, (mode == RBM_NORMAL_NO_READ), &found);
if (found)
pgBufferUsage.shared_blks_hit++;
+ else if (mode == RBM_NORMAL_NO_READ)
+ return InvalidBuffer;
else
pgBufferUsage.shared_blks_read++;
}
@@ -788,6 +793,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
return BufferDescriptorGetBuffer(bufHdr);
}
+ Assert(mode != RBM_NORMAL_NO_READ);
+
/*
* We get here only in the corner case where we are trying to extend
* the relation but we found a pre-existing buffer marked BM_VALID.
@@ -987,6 +994,7 @@ static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
+ bool onlyInMemory,
bool *foundPtr)
{
BufferTag newTag; /* identity of requested block */
@@ -1036,7 +1044,17 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* own read attempt if the page is still not BM_VALID.
* StartBufferIO does it all.
*/
- if (StartBufferIO(buf, true))
+ if (onlyInMemory == true)
+ {
+ /*
+ * We cannot get buffer immediately, go away if caller just
+ * tried to get access to already in-memory page
+ */
+ UnpinBuffer(buf, true);
+ *foundPtr = false;
+ return NULL;
+ }
+ else if (StartBufferIO(buf, true))
{
/*
* If we get here, previous attempts to read the buffer must
@@ -1055,6 +1073,13 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ /* will not read a page from disk */
+ if (onlyInMemory == true)
+ {
+ *foundPtr = false;
+ return NULL;
+ }
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1496,6 +1521,37 @@ MarkBufferDirty(Buffer buffer)
}
}
+/*
+ * IsBufferDirty - is buffer marked dirty?
+ *
+ * Buffer should be pinned, if caller wishes to get consistent result,
+ * buffer should be exclusively locked.
+ */
+bool
+IsBufferDirty(Buffer buffer)
+{
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ if (!BufferIsValid(buffer))
+ elog(ERROR, "bad buffer ID: %d", buffer);
+
+ Assert(BufferIsPinned(buffer));
+
+ if (BufferIsLocal(buffer))
+ return IsLocalBufferDirty(buffer);
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * if buffer is not exclusively locked, buf_state could be out of sync
+ * already...
+ */
+ return (buf_state & BM_DIRTY) ? true : false;
+}
+
/*
* ReleaseAndReadBuffer -- combine ReleaseBuffer() and ReadBuffer()
*
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index e4146a260a..856955635e 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -305,6 +305,30 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * IsLocalBufferDirty
+ * returns true of buffer is dirty
+ */
+bool
+IsLocalBufferDirty(Buffer buffer)
+{
+ int bufid;
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ Assert(BufferIsLocal(buffer));
+
+ bufid = -(buffer + 1);
+
+ Assert(LocalRefCount[bufid] > 0);
+
+ bufHdr = GetLocalBufferDescriptor(bufid);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ return (buf_state & BM_DIRTY) != 0;
+}
+
/*
* DropRelFileNodeLocalBuffers
* This function removes from the buffer pool all the pages of the
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 0c86a581c0..fba74752a9 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -25,6 +25,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
@@ -140,6 +141,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, ProcSignalShmemSize());
size = add_size(size, CheckpointerShmemSize());
size = add_size(size, AutoVacuumShmemSize());
+ size = add_size(size, HeapCleanerShmemSize());
size = add_size(size, ReplicationSlotsShmemSize());
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
@@ -256,6 +258,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
ProcSignalShmemInit();
CheckpointerShmemInit();
AutoVacuumShmemInit();
+ HeapCleanerShmemInit();
ReplicationSlotsShmemInit();
ReplicationOriginShmemInit();
WalSndShmemInit();
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index e6025ecedb..73a9d141b0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -50,3 +50,4 @@ OldSnapshotTimeMapLock 42
BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
+HeapCleanerLock 46
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 6f30e082b2..ecf14647b4 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
@@ -352,7 +353,7 @@ InitProcess(void)
* cleaning up. (XXX autovac launcher currently doesn't participate in
* this; it probably should.)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildActive();
/*
@@ -896,7 +897,7 @@ ProcKill(int code, Datum arg)
* way, so tell the postmaster we've cleaned up acceptably well. (XXX
* autovac launcher should be included here someday)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildInactive();
/* wake autovac launcher if needed -- see comments in FreeWorkerInfo */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f4133953be..87779bd178 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -56,6 +56,7 @@
#include "parser/parser.h"
#include "pg_getopt.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -2889,6 +2890,10 @@ ProcessInterrupts(void)
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating autovacuum process due to administrator command")));
+ else if (IsHeapCleanerWorkerProcess())
+ ereport(FATAL,
+ (errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("terminating bgheap process due to administrator command")));
else if (IsLogicalWorker())
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
@@ -3017,6 +3022,13 @@ ProcessInterrupts(void)
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("canceling autovacuum task")));
}
+ if (IsHeapCleanerWorkerProcess())
+ {
+ LockErrorCleanup();
+ ereport(ERROR,
+ (errcode(ERRCODE_QUERY_CANCELED),
+ errmsg("canceling bgheap worker task")));
+ }
if (RecoveryConflictPending)
{
RecoveryConflictPending = false;
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index e95e347184..8831d98460 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
cmdtype = PROGRESS_COMMAND_VACUUM;
+ else if (pg_strcasecmp(cmd, "CLEANER") == 0)
+ cmdtype = PROGRESS_COMMAND_CLEANER;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/backend/utils/hash/Makefile b/src/backend/utils/hash/Makefile
index 64eebd1d99..9a3392fe37 100644
--- a/src/backend/utils/hash/Makefile
+++ b/src/backend/utils/hash/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/hash
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = dynahash.o hashfn.o pg_crc.o
+OBJS = dynahash.o hashfn.o pg_crc.o shash.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/hash/shash.c b/src/backend/utils/hash/shash.c
new file mode 100644
index 0000000000..9710485c85
--- /dev/null
+++ b/src/backend/utils/hash/shash.c
@@ -0,0 +1,265 @@
+/*
+ * shash.c
+ *
+ * Lightweight hash tables without dynamic expansion
+ *
+ *
+ * Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/hash/shash.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "limits.h"
+#include "utils/shash.h"
+
+#define ELEM(index) (&shtab->Elements[index*shtab->Header.ElementSize])
+
+static void check(SHTAB* shtab);
+
+/*
+ * Check data allocation of hash table variables
+ */
+static void
+check(SHTAB* shtab)
+{
+ Assert(shtab != NULL);
+ Assert(shtab->Elements != NULL);
+ Assert(shtab->state != NULL);
+}
+
+/*
+ * Create and initialize hash table with shctl options
+ */
+SHTAB*
+SHASH_Create(SHTABCTL shctl)
+{
+ SHTAB* shtab = (SHTAB *) palloc(sizeof(SHTAB));
+
+ Assert(shctl.ElementSize != 0);
+ Assert(shctl.ElementSize >= shctl.KeySize);
+
+ shtab->Header = shctl;
+ shtab->HTableSize = shtab->Header.ElementsMaxNum * (2. - shtab->Header.FillFactor) + 1;
+ Assert(shtab->HTableSize > shtab->Header.ElementsMaxNum);
+
+ /* Add one element as sign of empty value */
+ shtab->Elements = (char *) palloc(shtab->HTableSize * shctl.ElementSize);
+ shtab->state = (HESTATE *) palloc0(shtab->HTableSize * sizeof(HESTATE));
+ shtab->nElements = 0;
+ shtab->SeqScanCurElem = 0;
+
+ return shtab;
+}
+
+/*
+ * Fast cleaning of hash table.
+ * After call of this function hash table contains 0 elements
+ * and can be used again.
+ */
+void
+SHASH_Clean(SHTAB* shtab)
+{
+ int i;
+
+ check(shtab);
+
+ /*
+ * We do not free memory: only clean 'used' field and set number of elems at 0
+ */
+ for (i = 0; i < shtab->HTableSize; i++)
+ shtab->state[i] = SHASH_NUSED;
+ shtab->nElements = 0;
+ shtab->SeqScanCurElem = 0;
+}
+
+/*
+ * Free all hash table memory resources.
+ */
+void
+SHASH_Destroy(SHTAB* shtab)
+{
+ check(shtab);
+
+ pfree(shtab->Elements);
+ pfree(shtab->state);
+ pfree(shtab);
+}
+
+/*
+ * Get number of elements in the hash table.
+ */
+uint64
+SHASH_Entries(SHTAB* shtab)
+{
+ check(shtab);
+
+ Assert((shtab->nElements >= 0) && (shtab->nElements <= shtab->Header.ElementsMaxNum));
+ return shtab->nElements;
+}
+
+/*
+ * Find, enter or delete element.
+ * return NULL, if element not found.
+ */
+void*
+SHASH_Search(SHTAB* shtab, void *keyPtr, SHASHACTION action, bool *foundPtr)
+{
+ uint64 index,
+ first,
+ first_removed_index = ULONG_MAX;
+ void *result = NULL;
+
+ check(shtab);
+
+ Assert(shtab->nElements <= shtab->Header.ElementsMaxNum);
+
+ first = index = shtab->Header.HashFunc(keyPtr, shtab->Header.KeySize, shtab->HTableSize);
+ Assert(index < shtab->HTableSize);
+
+ if (foundPtr != NULL)
+ *foundPtr = false;
+
+ /*
+ * Main hash table search cycle
+ */
+ for(;;)
+ {
+ if (shtab->state[index] == SHASH_NUSED)
+ {
+ /* Empty position found */
+ switch (action)
+ {
+ case SHASH_ENTER:
+ if (shtab->nElements < shtab->Header.ElementsMaxNum)
+ {
+ if (first_removed_index != ULONG_MAX)
+ index= first_removed_index;
+
+ memcpy(ELEM(index), keyPtr, shtab->Header.KeySize);
+ shtab->state[index] = SHASH_USED;
+ shtab->nElements++;
+ result = ELEM(index);
+ }
+ break;
+ case SHASH_FIND:
+ /* Hash table not contains record with such key */
+ case SHASH_REMOVE:
+ /* may be potential problem */
+ break;
+ default:
+ Assert(0);
+ }
+
+ return result;
+ }
+
+ if ((shtab->state[index] == SHASH_USED) && (shtab->Header.CompFunc(keyPtr, ELEM(index))))
+ {
+ /*
+ * We found the element exactly
+ */
+ if (foundPtr != NULL)
+ *foundPtr = true;
+
+ if (action == SHASH_REMOVE)
+ {
+ Assert(shtab->nElements > 0);
+ /* Data will be cleaned by ENTER operation */
+ shtab->state[index] = SHASH_REMOVED;
+ shtab->nElements--;
+ }
+
+ return ELEM(index);
+ }
+
+ if ((shtab->state[index] == SHASH_REMOVED) && (action == SHASH_ENTER))
+ /* We can use this element for SHASH_ENTER, potentially */
+ if (first_removed_index == ULONG_MAX)
+ first_removed_index = index;
+
+ /* Go to next element */
+ index = (index+1)%(shtab->HTableSize);
+
+ if (index == first)
+ {
+ /*
+ * We made all-table search cycle.
+ */
+ if ((action == SHASH_ENTER) && (first_removed_index != ULONG_MAX) && (shtab->nElements < shtab->Header.ElementsMaxNum))
+ {
+ index = first_removed_index;
+ memcpy(ELEM(index), keyPtr, shtab->Header.KeySize);
+ shtab->state[index] = SHASH_USED;
+ shtab->nElements++;
+ result = ELEM(index);
+ }
+ else
+ result = NULL;
+ return result;
+ }
+ }
+
+ Assert(0);
+ return NULL;
+}
+
+/*
+ * Begin new sequental scan of the hash table.
+ */
+void
+SHASH_SeqReset(SHTAB* shtab)
+{
+ check(shtab);
+
+ shtab->SeqScanCurElem = 0;
+}
+
+/*
+ * Get next hash table value during sequental scan.
+ * Call after last element returned returns NULL.
+ */
+void*
+SHASH_SeqNext(SHTAB* shtab)
+{
+ check(shtab);
+
+ if (shtab->SeqScanCurElem == shtab->HTableSize)
+ return NULL;
+
+ Assert(shtab->SeqScanCurElem < shtab->HTableSize);
+
+ do
+ {
+ if (shtab->state[shtab->SeqScanCurElem] == SHASH_USED)
+ return ELEM(shtab->SeqScanCurElem++);
+ else
+ shtab->SeqScanCurElem++;
+ }
+ while (shtab->SeqScanCurElem < shtab->HTableSize);
+
+ return NULL;
+}
+
+/*
+ * Simple hash function.
+ */
+uint64
+DefaultHashValueFunc(void *key, uint64 size, uint64 base)
+{
+ uint64 sum = 0;
+ int i;
+
+ for (i = 0; i < size; i++)
+ {
+ uint64 x = ((char *)key)[i];
+ sum += x*x*x*x + x*x*x + x*x + x + 1;
+ }
+
+ return sum%base;
+}
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 865119d272..9ebdd1c7de 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -689,7 +690,7 @@ InitializeSessionUserIdStandalone(void)
* This function should only be called in single-user mode, in autovacuum
* workers, and in background workers.
*/
- AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker);
+ AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker || IsHeapCleanerWorkerProcess());
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 5ef6315d20..4345b8bbbc 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
@@ -321,7 +322,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
*
* We do not enforce them for autovacuum worker processes either.
*/
- if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess() && !IsHeapCleanerWorkerProcess())
{
/*
* Check that the database is currently allowing connections.
@@ -504,8 +505,8 @@ InitializeMaxBackends(void)
Assert(MaxBackends == 0);
/* the extra unit accounts for the autovacuum launcher */
- MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
- max_worker_processes;
+ MaxBackends = MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 + max_worker_processes;
/* internal error because the values were all checked previously */
if (MaxBackends > MAX_BACKENDS)
@@ -682,7 +683,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
before_shmem_exit(ShutdownPostgres, 0);
/* The autovacuum launcher is done here */
- if (IsAutoVacuumLauncherProcess())
+ if (IsAutoVacuumLauncherProcess() || IsHeapCleanerLauncherProcess())
{
/* report this backend in the PgBackendStatus array */
pgstat_bestart();
@@ -722,7 +723,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
* In standalone mode and in autovacuum worker processes, we use a fixed
* ID, otherwise we figure it out from the authenticated user name.
*/
- if (bootstrap || IsAutoVacuumWorkerProcess())
+ if (bootstrap || IsAutoVacuumWorkerProcess() || IsHeapCleanerWorkerProcess())
{
InitializeSessionUserIdStandalone();
am_superuser = true;
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 6a6b467fee..0cd51ad315 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,12 @@
#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+/* Progress parameters for cleaner */
+#define PROGRESS_CLEANER_RELATIONS 0
+#define PROGRESS_CLEANER_TOTAL_QUEUE_LENGTH 1
+#define PROGRESS_CLEANER_FALSE_HITS 2
+#define PROGRESS_CLEANER_TOTAL_DELETIONS 3
+#define PROGRESS_CLEANER_VAINLY_CLEANED_TUPLES 4
+#define PROGRESS_CLEANER_MISSED_BLOCKS 5
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 85d472f0a5..705120eddc 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -186,6 +186,9 @@ extern void vacuum_set_xid_limits(Relation rel,
extern void vac_update_datfrozenxid(void);
extern void vacuum_delay_point(void);
+extern void quick_vacuum_index(Relation irel, Relation hrel,
+ ItemPointer dead_tuples,
+ int num_dead_tuples);
/* in commands/vacuumlazy.c */
extern void lazy_vacuum_rel(Relation onerel, int options,
VacuumParams *params, BufferAccessStrategy bstrategy);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d59c24ae23..dbf09328ac 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -56,6 +56,7 @@ typedef enum StatMsgType
PGSTAT_MTYPE_RESETSHAREDCOUNTER,
PGSTAT_MTYPE_RESETSINGLECOUNTER,
PGSTAT_MTYPE_AUTOVAC_START,
+ PGSTAT_MTYPE_BGHEAP_START,
PGSTAT_MTYPE_VACUUM,
PGSTAT_MTYPE_ANALYZE,
PGSTAT_MTYPE_ARCHIVER,
@@ -354,6 +355,12 @@ typedef struct PgStat_MsgAutovacStart
TimestampTz m_start_time;
} PgStat_MsgAutovacStart;
+typedef struct PgStat_MsgHeapCleanerStart
+{
+ PgStat_MsgHdr m_hdr;
+ Oid m_databaseid;
+ TimestampTz m_start_time;
+} PgStat_MsgHeapCleanerStart;
/* ----------
* PgStat_MsgVacuum Sent by the backend or autovacuum daemon
@@ -704,6 +711,8 @@ typedef enum BackendType
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
+ B_BG_HEAPCLNR_LAUNCHER,
+ B_BG_HEAPCLNR_WORKER,
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
@@ -756,6 +765,7 @@ typedef enum
{
WAIT_EVENT_ARCHIVER_MAIN = PG_WAIT_ACTIVITY,
WAIT_EVENT_AUTOVACUUM_MAIN,
+ WAIT_EVENT_BGHEAP_MAIN,
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
@@ -933,7 +943,8 @@ typedef enum
typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
- PROGRESS_COMMAND_VACUUM
+ PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_CLEANER
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 10
@@ -1181,6 +1192,7 @@ extern void pgstat_reset_shared_counters(const char *);
extern void pgstat_reset_single_counter(Oid objectid, PgStat_Single_Reset_Type type);
extern void pgstat_report_autovac(Oid dboid);
+extern void pgstat_report_heapcleaner(Oid dboid);
extern void pgstat_report_vacuum(Oid tableoid, bool shared,
PgStat_Counter livetuples, PgStat_Counter deadtuples);
extern void pgstat_report_analyze(Relation rel,
diff --git a/src/include/postmaster/bgheap.h b/src/include/postmaster/bgheap.h
new file mode 100644
index 0000000000..e11b310ffd
--- /dev/null
+++ b/src/include/postmaster/bgheap.h
@@ -0,0 +1,36 @@
+/*
+ * bgheap.h
+ *
+ * Created on: 27.06.2018
+ * Author: andrey
+ */
+
+#ifndef SRC_INCLUDE_ACCESS_BGHEAP_H_
+#define SRC_INCLUDE_ACCESS_BGHEAP_H_
+
+#include "storage/block.h"
+
+#ifdef EXEC_BACKEND
+extern void HeapCleanerLauncherIAm(void);
+extern void HeapCleanerWorkerIAm(void);
+extern void HeapCleanerLauncherMain(int argc, char *argv[]) pg_attribute_noreturn();
+extern void HeapCleanerWorkerMain(int argc, char *argv[]) pg_attribute_noreturn();
+#endif
+
+extern int heapcleaner_max_workers;
+
+
+extern void AtEOXact_BGHeap_tables(bool isCommit);
+extern bool IsHeapCleanerLauncherProcess(void);
+extern bool IsHeapCleanerWorkerProcess(void);
+
+extern void HeapCleanerInit(void);
+extern int StartHeapCleanerLauncher(void);
+extern int StartHeapCleanerWorker(void);
+
+extern void HeapCleanerSend(Relation relation, BlockNumber blkno);
+
+extern void HeapCleanerShmemInit(void);
+extern Size HeapCleanerShmemSize(void);
+
+#endif /* SRC_INCLUDE_ACCESS_BGHEAP_H_ */
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5370035f0c..58be48198b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -333,6 +333,7 @@ extern void LocalPrefetchBuffer(SMgrRelation smgr, ForkNumber forkNum,
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
BlockNumber blockNum, bool *foundPtr);
extern void MarkLocalBufferDirty(Buffer buffer);
+extern bool IsLocalBufferDirty(Buffer buffer);
extern void DropRelFileNodeLocalBuffers(RelFileNode rnode, ForkNumber forkNum,
BlockNumber firstDelBlock);
extern void DropRelFileNodeAllLocalBuffers(RelFileNode rnode);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 3cce3906a0..dbf3843053 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -43,8 +43,10 @@ typedef enum
RBM_ZERO_AND_CLEANUP_LOCK, /* Like RBM_ZERO_AND_LOCK, but locks the page
* in "cleanup" mode */
RBM_ZERO_ON_ERROR, /* Read, but return an all-zeros page on error */
- RBM_NORMAL_NO_LOG /* Don't log page as invalid during WAL
+ RBM_NORMAL_NO_LOG, /* Don't log page as invalid during WAL
* replay; otherwise same as RBM_NORMAL */
+ RBM_NORMAL_NO_READ /* Don't actually read page, returns only if it
+ presents in shared memory */
} ReadBufferMode;
/* forward declared, to avoid having to expose buf_internals.h here */
@@ -175,6 +177,7 @@ extern Buffer ReadBufferWithoutRelcache(RelFileNode rnode,
extern void ReleaseBuffer(Buffer buffer);
extern void UnlockReleaseBuffer(Buffer buffer);
extern void MarkBufferDirty(Buffer buffer);
+extern bool IsBufferDirty(Buffer buffer);
extern void IncrBufferRefCount(Buffer buffer);
extern Buffer ReleaseAndReadBuffer(Buffer buffer, Relation relation,
BlockNumber blockNum);
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 5ecc1b757c..4c73acd62c 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -36,6 +36,8 @@ typedef enum
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
+ PMSIGNAL_START_HEAPCLNR_LAUNCHER,
+ PMSIGNAL_START_HEAPCLNR_WORKER,
PMSIGNAL_BACKGROUND_WORKER_CHANGE, /* background worker state change */
PMSIGNAL_START_WALRECEIVER, /* start a walreceiver */
PMSIGNAL_ADVANCE_STATE_MACHINE, /* advance postmaster's state machine */
diff --git a/src/include/utils/shash.h b/src/include/utils/shash.h
new file mode 100644
index 0000000000..7a209cf5dc
--- /dev/null
+++ b/src/include/utils/shash.h
@@ -0,0 +1,54 @@
+#ifndef SHASH_H_
+#define SHASH_H_
+
+/* hash_search operations */
+typedef enum
+{
+ SHASH_FIND,
+ SHASH_ENTER,
+ SHASH_REMOVE
+} SHASHACTION;
+
+/* States of a hash table element */
+typedef enum
+{
+ SHASH_NUSED = 0,
+ SHASH_USED,
+ SHASH_REMOVED
+} HESTATE;
+
+typedef uint64 (*SHashValueFunc) (void *key, uint64 size, uint64 base);
+typedef bool (*CompareFunc) (void* bucket1, void* bucket2);
+
+typedef struct SHTABCTL
+{
+ uint64 ElementSize;
+ uint64 KeySize;
+ uint64 ElementsMaxNum;
+ float FillFactor;
+ SHashValueFunc HashFunc;
+ CompareFunc CompFunc;
+} SHTABCTL;
+
+typedef struct SHTAB
+{
+ SHTABCTL Header;
+ char *Elements;
+ uint64 nElements;
+ HESTATE *state; /* State of an element of hash table*/
+ uint64 SeqScanCurElem;
+ uint64 HTableSize;
+} SHTAB;
+
+typedef struct SHTAB* PSHTAB;
+
+extern PSHTAB SHASH_Create(SHTABCTL shctl);
+extern void SHASH_Clean(PSHTAB shtab);
+extern void SHASH_Destroy(PSHTAB shtab);
+extern uint64 SHASH_Entries(PSHTAB shtab);
+extern void SHASH_SeqReset(PSHTAB shtab);
+extern void* SHASH_SeqNext(PSHTAB shtab);
+extern void* SHASH_Search(PSHTAB shtab, void *keyPtr, SHASHACTION action, bool *foundPtr);
+extern uint64 DefaultHashValueFunc(void *key, uint64 size, uint64 base);
+
+#endif /* SHASH_H_ */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..b94ce20f68 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -167,7 +167,7 @@ select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
a | b
---+----
1 | 11
@@ -1821,6 +1821,18 @@ pg_stat_database_conflicts| SELECT d.oid AS datid,
pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin,
pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock
FROM pg_database d;
+pg_stat_progress_cleaner| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.relid,
+ s.param1 AS relations,
+ s.param2 AS queue_len,
+ s.param3 AS false_hits,
+ s.param4 AS cleaned_tuples,
+ s.param5 AS vainly_cleaned_tuples,
+ s.param6 AS missed_tuples
+ FROM (pg_stat_get_progress_info('CLEANER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index e57c6e1c42..b9c0af7bce 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -637,7 +637,7 @@ select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
i
---
3
diff --git a/src/test/regress/input/constraints.source b/src/test/regress/input/constraints.source
index 98dd4210e9..2968a1230b 100644
--- a/src/test/regress/input/constraints.source
+++ b/src/test/regress/input/constraints.source
@@ -337,7 +337,7 @@ ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- explicitly defer the constraint
BEGIN;
@@ -349,7 +349,7 @@ DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- try adding an initially deferred constraint
ALTER TABLE unique_tbl DROP CONSTRAINT unique_tbl_i_key;
@@ -367,7 +367,7 @@ DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- should fail at commit-time
BEGIN;
@@ -420,7 +420,7 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- test a HOT update that modifies the newly inserted tuple,
-- but should succeed because we then remove the other conflicting tuple.
@@ -431,11 +431,11 @@ INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
DROP TABLE unique_tbl;
diff --git a/src/test/regress/output/constraints.source b/src/test/regress/output/constraints.source
index a6a1df18e7..a2ae6df587 100644
--- a/src/test/regress/output/constraints.source
+++ b/src/test/regress/output/constraints.source
@@ -478,7 +478,7 @@ DETAIL: Key (i)=(1) already exists.
ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+------
1 | one
@@ -494,14 +494,14 @@ SET CONSTRAINTS unique_tbl_i_key DEFERRED;
INSERT INTO unique_tbl VALUES (3, 'three');
DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
1 | one
2 | two
+ 3 | three
4 | four
5 | five
- 3 | three
(5 rows)
-- try adding an initially deferred constraint
@@ -516,14 +516,14 @@ UPDATE unique_tbl SET i = 2 WHERE i = 4 AND t = 'four';
DELETE FROM unique_tbl WHERE i = 1 AND t = 'one';
DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- should fail at commit-time
@@ -581,14 +581,14 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
ERROR: duplicate key value violates unique constraint "unique_tbl_i_key"
DETAIL: Key (i)=(3) already exists.
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- test a HOT update that modifies the newly inserted tuple,
@@ -597,25 +597,25 @@ BEGIN;
INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
DROP TABLE unique_tbl;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 3ca4c07356..c540f12a29 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -181,7 +181,7 @@ delete from rtest_v1 where b = 12;
select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
** Remember the delete rule on rtest_v1: It says
** DO INSTEAD DELETE FROM rtest_t1 WHERE a = old.a
** So this time both rows with a = 2 must get deleted
diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql
index 1250cd63e0..5a45fbad41 100644
--- a/src/test/regress/sql/triggers.sql
+++ b/src/test/regress/sql/triggers.sql
@@ -419,7 +419,7 @@ delete from trigtest where i=1;
select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
drop table trigtest2;
drop table trigtest;
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-09-05 12:25 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-06 10:26 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
3 siblings, 1 reply; 47+ messages in thread
From: Andrey Lepikhov @ 2018-09-05 12:25 UTC (permalink / raw)
To: pgsql-hackers
Hi,
I prepared next version of Background worker (cleaner) based on a retail
indextuple deletion patch.
This version shows stable behavior on regression tests and pgbench
workloads.
In this version:
1. Only AccessShareLock are acquired on a cleanup of heap and index
relations.
2. Some 'aggressive' cleanup strategy introduced - conditional cleanup
locks not used.
3. Cleanup only an in-memory blocks.
4. The Cleaner calls heap_page_prune() before cleanup a block.
Benchmarks
---------
Two factors were evaluated: performance (tps) and relations blowing.
Before each test some rarefaction of pgbench_accounts was modeled by
deletion 10% of tuples at each block.
Also, I tested normal and Gaussian distribution of queries on
pgbench_accounts relation.
Autovacuum uses default settings.
Script:
pgbench -i -s 10
psql -c $"DELETE FROM pgbench_accounts WHERE (random() < 0.1);"
psql -c $"VACUUM;"
psql -c $"CREATE INDEX pgbench_accounts_ext ON public.pgbench_accounts
USING btree (abalance);" &&
pgbench -T 3600 -c 32 -j 8 -M prepared -P 600
NORMAL distribution:
average tps = 1045 (cleaner); = 1077 (autovacuum)
Relations size at the end of test, MB:
pgbench_accounts: 128 (cleaner); 128 (autovacuum)
pgbench_branches: 0.1 (cleaner); 2.1 (autovacuum)
pgbench_tellers: 0.4 (cleaner); 2.8 (autovacuum)
pgbench_accounts_pkey: 21 (cleaner); 43 (autovacuum)
pgbench_accounts_ext: 48 (cleaner); 56 (autovacuum)
Gaussian distribution:
average tps = 213 (cleaner); = 213 (autovacuum)
Relations size at the end of test, MB:
pgbench_accounts: 128 (cleaner); 128 (autovacuum)
pgbench_accounts_ext: 22 (cleaner); 29 (autovacuum)
Conclusions
-----------
1. For retail indextuple deletion purposes i replaced ItemIdSetDead() by
ItemIdMarkDead() in heap_page_prune_execute() operation. Hereupon in the
case of 100% filling of each relation block we get a blowing HEAP and
index , more or less. When the blocks already have free space, the
cleaner can delay blowing the heap and index without a vacuum.
2. Cleaner works fine in the case of skewness of access frequency to
relation blocks.
3. The cleaner does not cause a decrease of performance.
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments:
0005-Heap-and-Index-cleaner.patch (text/x-patch, 54.0K) download
From b465357b8a884cac3b503ad171976d3afd31f574 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Wed, 5 Sep 2018 10:39:11 +0300
Subject: [PATCH 5/5] Heap and Index cleaner
---
.../postgres_fdw/expected/postgres_fdw.out | 30 ++--
contrib/postgres_fdw/sql/postgres_fdw.sql | 4 +-
src/backend/access/heap/pruneheap.c | 6 +-
src/backend/access/nbtree/nbtree.c | 15 +-
src/backend/access/transam/xact.c | 4 +
src/backend/catalog/system_views.sql | 11 ++
src/backend/commands/vacuumlazy.c | 44 +++--
src/backend/postmaster/Makefile | 2 +-
src/backend/postmaster/pgstat.c | 36 ++++
src/backend/postmaster/postmaster.c | 160 +++++++++++++++++-
src/backend/storage/buffer/bufmgr.c | 60 ++++++-
src/backend/storage/buffer/localbuf.c | 24 +++
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/storage/lmgr/proc.c | 5 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/backend/utils/hash/Makefile | 2 +-
src/backend/utils/init/miscinit.c | 3 +-
src/backend/utils/init/postinit.c | 11 +-
src/include/commands/progress.h | 12 ++
src/include/commands/vacuum.h | 3 +
src/include/pgstat.h | 14 +-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 5 +-
src/include/storage/pmsignal.h | 2 +
src/test/regress/expected/rules.out | 18 +-
src/test/regress/expected/triggers.out | 2 +-
src/test/regress/input/constraints.source | 12 +-
src/test/regress/output/constraints.source | 34 ++--
src/test/regress/sql/rules.sql | 2 +-
src/test/regress/sql/triggers.sql | 2 +-
32 files changed, 458 insertions(+), 84 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index f5498c62bd..622bea2a7d 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -5496,19 +5496,25 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
- QUERY PLAN
-----------------------------------------------------------------------------------------------------------
- Update on public.ft2
- Output: c1, c2, c3, c4, c5, c6, c7, c8
- Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
- -> Foreign Scan on public.ft2
- Output: c1, c2, NULL::integer, 'bar'::text, c4, c5, c6, c7, c8, ctid
- Filter: (postgres_fdw_abs(ft2.c1) > 2000)
- Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
-(7 rows)
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------
+ Sort
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+ Sort Key: updated.c1
+ CTE updated
+ -> Update on public.ft2
+ Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: ft2.c1, ft2.c2, NULL::integer, 'bar'::text, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.ctid
+ Filter: (postgres_fdw_abs(ft2.c1) > 2000)
+ Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
+ -> CTE Scan on updated
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+(13 rows)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
------+----+-----+----+----+----+------------+----
2001 | 1 | bar | | | | ft2 |
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index e1b955f3f0..6034e83108 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1137,8 +1137,8 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c3 = 'baz'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index ebbafe3275..e7c2d897f3 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -22,6 +22,7 @@
#include "catalog/catalog.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
@@ -156,6 +157,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/* OK to prune */
(void) heap_page_prune(relation, buffer, OldestXmin, true, &ignore);
+ if (IsBufferDirty(buffer))
+ HeapCleanerSend(relation, BufferGetBlockNumber(buffer));
}
/* And release buffer lock */
@@ -411,7 +414,6 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
- ndeleted++;
}
/* Nothing more to do */
@@ -583,9 +585,9 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- ndeleted++;
if (chainitems[i] == latestdead)
continue;
+ ndeleted++;
heap_prune_record_unused(prstate, chainitems[i]);
}
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 50378e0f51..28650008ef 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -930,7 +930,14 @@ bttargetdelete(IndexTargetDeleteInfo *info,
skey = _bt_mkscankey(irel, itup);
/* Descend the tree and position ourselves on the target leaf page. */
- stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_READ, NULL);
+ stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_WRITE, NULL);
+
+ /* trade in our read lock for a write lock */
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBuffer(buf, BT_WRITE);
+
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_WRITE, NULL);
/* To prepare tuple entries search across index pages */
Assert(BufferIsValid(buf));
@@ -954,6 +961,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
@@ -974,7 +982,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
* Traverse to a next reliable index page
*/
buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
- false, true, stack, BT_READ, NULL);
+ false, true, stack, BT_WRITE, NULL);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
@@ -1019,6 +1027,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
}
else if (res > 0)
break;
+
pos--;
}
@@ -1033,11 +1042,11 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
_bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
-
stats->tuples_removed += ndeletable;
}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9aa63c8792..f5e70ae16d 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -40,6 +40,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -2126,6 +2127,7 @@ CommitTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(true);
AtEOXact_Snapshot(true, false);
AtEOXact_ApplyLauncher(true);
@@ -2404,6 +2406,7 @@ PrepareTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
/* don't call AtEOXact_PgStat here; we fixed pgstat state above */
AtEOXact_Snapshot(true, true);
pgstat_report_xact_timestamp(0);
@@ -2606,6 +2609,7 @@ AbortTransaction(void)
AtEOXact_Files(false);
AtEOXact_ComboCid();
AtEOXact_HashTables(false);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(false);
AtEOXact_ApplyLauncher(false);
pgstat_report_xact_timestamp(0);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 7251552419..3b6e9aa41c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -903,6 +903,17 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_cleaner AS
+ SELECT
+ S.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid,
+ S.param1 AS relations, S.param2 AS queue_len, S.param3 AS ninmem,
+ S.param4 AS cleaned_tuples, S.param5 AS vain_tuples,
+ S.param6 AS missed_blocks, S.param7 AS cleaned_blocks,
+ S.param8 AS vain_blocks, S.param9 AS nlocks, S.param10 AS timeout
+ FROM pg_stat_get_progress_info('CLEANER') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid;
+
CREATE VIEW pg_user_mappings AS
SELECT
U.oid AS umid,
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 1b952fedd9..9b101e79fb 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -158,8 +158,6 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
-static void quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -742,7 +740,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1394,7 +1392,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1730,7 +1728,12 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
{
offnum = ItemIdGetRedirect(lp);
lp = PageGetItemId(page, offnum);
- Assert(ItemIdIsUsed(lp));
+
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
}
/* Form a tuple */
@@ -1747,30 +1750,34 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
* quick_vacuum_index() -- quick vacuum one index relation.
*
* Delete all the index entries pointing to tuples listed in
- * vacrelstats->dead_tuples.
+ * dead_tuples.
*/
-static void
+void
quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats)
+ ItemPointer dead_tuples,
+ int num_dead_tuples)
{
int tnum;
- bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
- IndexInfo* indexInfo = BuildIndexInfo(irel);
- EState* estate = CreateExecutorState();
- ExprContext* econtext = GetPerTupleExprContext(estate);
- ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
- TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ bool *found = palloc0(num_dead_tuples*sizeof(bool));
+ IndexInfo *indexInfo = BuildIndexInfo(irel);
+ EState *estate = CreateExecutorState();
+ ExprContext *econtext = GetPerTupleExprContext(estate);
+ ExprState *predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
IndexTargetDeleteResult stats;
IndexTargetDeleteInfo ivinfo;
+ Assert(found != NULL);
+
+ stats.tuples_removed = 0;
ivinfo.indexRelation = irel;
ivinfo.heapRelation = hrel;
econtext->ecxt_scantuple = slot;
/* Get tuple from heap */
- for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
+ for (tnum = num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1781,7 +1788,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
continue;
/* Get a tuple from heap */
- if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ if ((tuple = get_tuple_by_tid(hrel, &(dead_tuples[tnum]))) == NULL)
{
/*
* Tuple has 'not used' status.
@@ -1814,15 +1821,16 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.dead_tuples = dead_tuples;
ivinfo.last_dead_tuple = tnum;
ivinfo.found_dead_tuples = found;
+
index_target_delete(&ivinfo, &stats, values, isnull);
}
- pfree(found);
ExecDropSingleTupleTableSlot(slot);
FreeExecutorState(estate);
+ pfree(found);
}
/*
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c23211b2..11796375e8 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
+OBJS = autovacuum.o bgheap.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index bbe73618c7..f992307e37 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -48,6 +48,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/fork_process.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
@@ -1394,6 +1395,20 @@ pgstat_report_autovac(Oid dboid)
pgstat_send(&msg, sizeof(msg));
}
+void
+pgstat_report_heapcleaner(Oid dboid)
+{
+ PgStat_MsgAutovacStart msg;
+
+ if (pgStatSock == PGINVALID_SOCKET)
+ return;
+
+ pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_BGHEAP_START);
+ msg.m_databaseid = dboid;
+ msg.m_start_time = GetCurrentTimestamp();
+
+ pgstat_send(&msg, sizeof(msg));
+}
/* ---------
* pgstat_report_vacuum() -
@@ -2833,6 +2848,14 @@ pgstat_bestart(void)
/* Autovacuum Worker */
beentry->st_backendType = B_AUTOVAC_WORKER;
}
+ else if (IsHeapCleanerLauncherProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_LAUNCHER;
+ }
+ else if (IsHeapCleanerWorkerProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_WORKER;
+ }
else if (am_walsender)
{
/* Wal sender */
@@ -3483,6 +3506,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_AUTOVACUUM_MAIN:
event_name = "AutoVacuumMain";
break;
+ case WAIT_EVENT_BGHEAP_MAIN:
+ event_name = "BgHeapMain";
+ break;
case WAIT_EVENT_BGWRITER_HIBERNATE:
event_name = "BgWriterHibernate";
break;
@@ -4111,6 +4137,12 @@ pgstat_get_backend_desc(BackendType backendType)
case B_AUTOVAC_WORKER:
backendDesc = "autovacuum worker";
break;
+ case B_BG_HEAPCLNR_LAUNCHER:
+ backendDesc = "heap cleaner launcher";
+ break;
+ case B_BG_HEAPCLNR_WORKER:
+ backendDesc = "heap cleaner worker";
+ break;
case B_BACKEND:
backendDesc = "client backend";
break;
@@ -4422,6 +4454,10 @@ PgstatCollectorMain(int argc, char *argv[])
pgstat_recv_autovac((PgStat_MsgAutovacStart *) &msg, len);
break;
+ case PGSTAT_MTYPE_BGHEAP_START:
+// pgstat_recv_autovac((PgStat_MsgHeapCleanerStart *) &msg, len);
+ break;
+
case PGSTAT_MTYPE_VACUUM:
pgstat_recv_vacuum((PgStat_MsgVacuum *) &msg, len);
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b33cd..d14c1b557d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -109,6 +109,7 @@
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
@@ -145,9 +146,10 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_BGHEAP 0x0010 /* autovacuum worker process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
-#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
+#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP)
/*
* List of active backends (or child processes anyway; we don't actually
@@ -252,6 +254,7 @@ static pid_t StartupPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
+ HeapClnrPID = 0,
PgArchPID = 0,
PgStatPID = 0,
SysLoggerPID = 0;
@@ -355,6 +358,7 @@ bool redirection_done = false; /* stderr redirected for syslogger? */
/* received START_AUTOVAC_LAUNCHER signal */
static volatile sig_atomic_t start_autovac_launcher = false;
+static volatile sig_atomic_t start_heapclnr_launcher = false;
/* the launcher needs to be signalled to communicate some condition */
static volatile bool avlauncher_needs_signal = false;
@@ -432,6 +436,7 @@ static void maybe_start_bgworkers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
+static void StartBgHeapWorker(void);
static void MaybeStartWalReceiver(void);
static void InitPostmasterDeathWatchHandle(void);
@@ -1310,6 +1315,7 @@ PostmasterMain(int argc, char *argv[])
*/
autovac_init();
+ HeapCleanerInit();
/*
* Load configuration files for client authentication.
*/
@@ -1757,6 +1763,14 @@ ServerLoop(void)
start_autovac_launcher = false; /* signal processed */
}
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && start_heapclnr_launcher &&
+ pmState == PM_RUN)
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ if (HeapClnrPID != 0)
+ start_heapclnr_launcher = false; /* signal processed */
+ }
+
/* If we have lost the stats collector, try to start a new one */
if (PgStatPID == 0 &&
(pmState == PM_RUN || pmState == PM_HOT_STANDBY))
@@ -1774,6 +1788,12 @@ ServerLoop(void)
kill(AutoVacPID, SIGUSR2);
}
+ /* If we have lost the background heap cleaner, try to start a new one. */
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && pmState == PM_RUN )
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ }
+
/* If we need to start a WAL receiver, try to do that now */
if (WalReceiverRequested)
MaybeStartWalReceiver();
@@ -2541,6 +2561,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGHUP);
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGHUP);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGHUP);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGHUP);
if (SysLoggerPID != 0)
@@ -2627,10 +2649,12 @@ pmdie(SIGNAL_ARGS)
/* autovac workers are told to shut down immediately */
/* and bgworkers too; does this need tweaking? */
SignalSomeChildren(SIGTERM,
- BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the bgwriter too */
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
@@ -2687,7 +2711,7 @@ pmdie(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGTERM);
if (pmState == PM_RECOVERY)
{
- SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER);
+ SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/*
* Only startup, bgwriter, walreceiver, possibly bgworkers,
@@ -2708,10 +2732,12 @@ pmdie(SIGNAL_ARGS)
/* shut down all backends and workers */
SignalSomeChildren(SIGTERM,
BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC |
- BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the walwriter too */
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGTERM);
@@ -2875,6 +2901,8 @@ reaper(SIGNAL_ARGS)
*/
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
+ if (!IsBinaryUpgrade && (HeapClnrPID == 0))
+ HeapClnrPID = StartHeapCleanerLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
PgArchPID = pgarch_start();
if (PgStatPID == 0)
@@ -3009,6 +3037,14 @@ reaper(SIGNAL_ARGS)
_("autovacuum launcher process"));
continue;
}
+ if (pid == HeapClnrPID)
+ {
+ HeapClnrPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("heap cleaner launcher process"));
+ continue;
+ }
/*
* Was it the archiver? If so, just try to start a new one; no need
@@ -3475,6 +3511,17 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ if (pid == HeapClnrPID)
+ HeapClnrPID = 0;
+ else if (HeapClnrPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ "SIGQUIT",
+ (int) HeapClnrPID)));
+ signal_child(HeapClnrPID, SIGQUIT);
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3661,6 +3708,7 @@ PostmasterStateMachine(void)
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
WalWriterPID == 0 &&
+ HeapClnrPID == 0 &&
AutoVacPID == 0)
{
if (Shutdown >= ImmediateShutdown || FatalError)
@@ -3759,6 +3807,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(HeapClnrPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -3948,6 +3997,8 @@ TerminateChildren(int signal)
signal_child(WalReceiverPID, signal);
if (AutoVacPID != 0)
signal_child(AutoVacPID, signal);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, signal);
if (PgArchPID != 0)
signal_child(PgArchPID, signal);
if (PgStatPID != 0)
@@ -4781,6 +4832,7 @@ SubPostmasterMain(int argc, char *argv[])
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
+ strcmp(argv[1], "--forkbgheap") == 0 ||
strcmp(argv[1], "--forkboot") == 0 ||
strncmp(argv[1], "--forkbgworker=", 15) == 0)
PGSharedMemoryReAttach();
@@ -4792,6 +4844,8 @@ SubPostmasterMain(int argc, char *argv[])
AutovacuumLauncherIAm();
if (strcmp(argv[1], "--forkavworker") == 0)
AutovacuumWorkerIAm();
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ BgHeapCleanerIAm();
/*
* Start our win32 signal implementation. This has to be done after we
@@ -4921,6 +4975,19 @@ SubPostmasterMain(int argc, char *argv[])
AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores(false, 0);
+
+ BgHeapMain(argc, argv); /* does not return */
+ }
if (strncmp(argv[1], "--forkbgworker=", 15) == 0)
{
int shmem_slot;
@@ -5122,6 +5189,12 @@ sigusr1_handler(SIGNAL_ARGS)
start_autovac_launcher = true;
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_LAUNCHER) &&
+ Shutdown == NoShutdown)
+ {
+ start_heapclnr_launcher = true;
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_AUTOVAC_WORKER) &&
Shutdown == NoShutdown)
{
@@ -5129,6 +5202,13 @@ sigusr1_handler(SIGNAL_ARGS)
StartAutovacuumWorker();
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_WORKER) &&
+ Shutdown == NoShutdown)
+ {
+ /* The autovacuum launcher wants us to start a worker process. */
+ StartBgHeapWorker();
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_WALRECEIVER))
{
/* Startup Process wants us to start the walreceiver process. */
@@ -5470,6 +5550,73 @@ StartAutovacuumWorker(void)
}
}
+static void
+StartBgHeapWorker(void)
+{
+ Backend *bn;
+
+ /*
+ * If not in condition to run a process, don't try, but handle it like a
+ * fork failure. This does not normally happen, since the signal is only
+ * supposed to be sent by autovacuum launcher when it's OK to do it, but
+ * we have to check to avoid race-condition problems during DB state
+ * changes.
+ */
+ if (canAcceptConnections() == CAC_OK)
+ {
+ /*
+ * Compute the cancel key that will be assigned to this session. We
+ * probably don't need cancel keys for autovac workers, but we'd
+ * better have something random in the field to prevent unfriendly
+ * people from sending cancels to them.
+ */
+ if (!RandomCancelKey(&MyCancelKey))
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random cancel key")));
+ return;
+ }
+
+ bn = (Backend *) malloc(sizeof(Backend));
+ if (bn)
+ {
+ bn->cancel_key = MyCancelKey;
+
+ /* Autovac workers are not dead_end and need a child slot */
+ bn->dead_end = false;
+ bn->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+ bn->bgworker_notify = false;
+
+ bn->pid = StartHeapCleanerWorker();
+
+ if (bn->pid > 0)
+ {
+ bn->bkend_type = BACKEND_TYPE_BGHEAP;
+ dlist_push_head(&BackendList, &bn->elem);
+#ifdef EXEC_BACKEND
+ ShmemBackendArrayAdd(bn);
+#endif
+ /* all OK */
+ return;
+ }
+
+ /*
+ * fork failed, fall through to report -- actual error message was
+ * logged by StartAutoVacWorker
+ */
+ (void) ReleasePostmasterChildSlot(bn->child_slot);
+ free(bn);
+ }
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+ }
+
+ elog(ERROR, "Some BGHEAP Worker start error!");
+}
+
/*
* MaybeStartWalReceiver
* Start the WAL receiver process, if not running and our state allows.
@@ -5534,7 +5681,8 @@ CreateOptsFile(int argc, char *argv[], char *fullprogname)
int
MaxLivePostmasterChildren(void)
{
- return 2 * (MaxConnections + autovacuum_max_workers + 1 +
+ return 2 * (MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 +
max_worker_processes);
}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 01eabe5706..175902e0d3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -449,6 +449,7 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
+ bool onlyInMemory,
bool *foundPtr);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
static void AtProcExit_Buffers(int code, Datum arg);
@@ -717,6 +718,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
isExtend = (blockNum == P_NEW);
+ Assert(!(isExtend && mode == RBM_NORMAL_NO_READ));
+
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
smgr->smgr_rnode.node.spcNode,
smgr->smgr_rnode.node.dbNode,
@@ -743,9 +746,11 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, (mode == RBM_NORMAL_NO_READ), &found);
if (found)
pgBufferUsage.shared_blks_hit++;
+ else if (mode == RBM_NORMAL_NO_READ)
+ return InvalidBuffer;
else
pgBufferUsage.shared_blks_read++;
}
@@ -788,6 +793,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
return BufferDescriptorGetBuffer(bufHdr);
}
+ Assert(mode != RBM_NORMAL_NO_READ);
+
/*
* We get here only in the corner case where we are trying to extend
* the relation but we found a pre-existing buffer marked BM_VALID.
@@ -987,6 +994,7 @@ static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
+ bool onlyInMemory,
bool *foundPtr)
{
BufferTag newTag; /* identity of requested block */
@@ -1036,7 +1044,17 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* own read attempt if the page is still not BM_VALID.
* StartBufferIO does it all.
*/
- if (StartBufferIO(buf, true))
+ if (onlyInMemory == true)
+ {
+ /*
+ * We cannot get buffer immediately, go away if caller just
+ * tried to get access to already in-memory page
+ */
+ UnpinBuffer(buf, true);
+ *foundPtr = false;
+ return NULL;
+ }
+ else if (StartBufferIO(buf, true))
{
/*
* If we get here, previous attempts to read the buffer must
@@ -1055,6 +1073,13 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ /* will not read a page from disk */
+ if (onlyInMemory == true)
+ {
+ *foundPtr = false;
+ return NULL;
+ }
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1496,6 +1521,37 @@ MarkBufferDirty(Buffer buffer)
}
}
+/*
+ * IsBufferDirty - is buffer marked dirty?
+ *
+ * Buffer should be pinned, if caller wishes to get consistent result,
+ * buffer should be exclusively locked.
+ */
+bool
+IsBufferDirty(Buffer buffer)
+{
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ if (!BufferIsValid(buffer))
+ elog(ERROR, "bad buffer ID: %d", buffer);
+
+ Assert(BufferIsPinned(buffer));
+
+ if (BufferIsLocal(buffer))
+ return IsLocalBufferDirty(buffer);
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * if buffer is not exclusively locked, buf_state could be out of sync
+ * already...
+ */
+ return (buf_state & BM_DIRTY) ? true : false;
+}
+
/*
* ReleaseAndReadBuffer -- combine ReleaseBuffer() and ReadBuffer()
*
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index e4146a260a..856955635e 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -305,6 +305,30 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * IsLocalBufferDirty
+ * returns true of buffer is dirty
+ */
+bool
+IsLocalBufferDirty(Buffer buffer)
+{
+ int bufid;
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ Assert(BufferIsLocal(buffer));
+
+ bufid = -(buffer + 1);
+
+ Assert(LocalRefCount[bufid] > 0);
+
+ bufHdr = GetLocalBufferDescriptor(bufid);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ return (buf_state & BM_DIRTY) != 0;
+}
+
/*
* DropRelFileNodeLocalBuffers
* This function removes from the buffer pool all the pages of the
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 0c86a581c0..fba74752a9 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -25,6 +25,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
@@ -140,6 +141,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, ProcSignalShmemSize());
size = add_size(size, CheckpointerShmemSize());
size = add_size(size, AutoVacuumShmemSize());
+ size = add_size(size, HeapCleanerShmemSize());
size = add_size(size, ReplicationSlotsShmemSize());
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
@@ -256,6 +258,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
ProcSignalShmemInit();
CheckpointerShmemInit();
AutoVacuumShmemInit();
+ HeapCleanerShmemInit();
ReplicationSlotsShmemInit();
ReplicationOriginShmemInit();
WalSndShmemInit();
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index e6025ecedb..73a9d141b0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -50,3 +50,4 @@ OldSnapshotTimeMapLock 42
BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
+HeapCleanerLock 46
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 6f30e082b2..ecf14647b4 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
@@ -352,7 +353,7 @@ InitProcess(void)
* cleaning up. (XXX autovac launcher currently doesn't participate in
* this; it probably should.)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildActive();
/*
@@ -896,7 +897,7 @@ ProcKill(int code, Datum arg)
* way, so tell the postmaster we've cleaned up acceptably well. (XXX
* autovac launcher should be included here someday)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildInactive();
/* wake autovac launcher if needed -- see comments in FreeWorkerInfo */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f4133953be..87779bd178 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -56,6 +56,7 @@
#include "parser/parser.h"
#include "pg_getopt.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -2889,6 +2890,10 @@ ProcessInterrupts(void)
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating autovacuum process due to administrator command")));
+ else if (IsHeapCleanerWorkerProcess())
+ ereport(FATAL,
+ (errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("terminating bgheap process due to administrator command")));
else if (IsLogicalWorker())
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
@@ -3017,6 +3022,13 @@ ProcessInterrupts(void)
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("canceling autovacuum task")));
}
+ if (IsHeapCleanerWorkerProcess())
+ {
+ LockErrorCleanup();
+ ereport(ERROR,
+ (errcode(ERRCODE_QUERY_CANCELED),
+ errmsg("canceling bgheap worker task")));
+ }
if (RecoveryConflictPending)
{
RecoveryConflictPending = false;
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index e95e347184..8831d98460 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
cmdtype = PROGRESS_COMMAND_VACUUM;
+ else if (pg_strcasecmp(cmd, "CLEANER") == 0)
+ cmdtype = PROGRESS_COMMAND_CLEANER;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/backend/utils/hash/Makefile b/src/backend/utils/hash/Makefile
index 64eebd1d99..9a3392fe37 100644
--- a/src/backend/utils/hash/Makefile
+++ b/src/backend/utils/hash/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/hash
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = dynahash.o hashfn.o pg_crc.o
+OBJS = dynahash.o hashfn.o pg_crc.o shash.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 865119d272..9ebdd1c7de 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -689,7 +690,7 @@ InitializeSessionUserIdStandalone(void)
* This function should only be called in single-user mode, in autovacuum
* workers, and in background workers.
*/
- AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker);
+ AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker || IsHeapCleanerWorkerProcess());
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 5ef6315d20..4345b8bbbc 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
@@ -321,7 +322,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
*
* We do not enforce them for autovacuum worker processes either.
*/
- if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess() && !IsHeapCleanerWorkerProcess())
{
/*
* Check that the database is currently allowing connections.
@@ -504,8 +505,8 @@ InitializeMaxBackends(void)
Assert(MaxBackends == 0);
/* the extra unit accounts for the autovacuum launcher */
- MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
- max_worker_processes;
+ MaxBackends = MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 + max_worker_processes;
/* internal error because the values were all checked previously */
if (MaxBackends > MAX_BACKENDS)
@@ -682,7 +683,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
before_shmem_exit(ShutdownPostgres, 0);
/* The autovacuum launcher is done here */
- if (IsAutoVacuumLauncherProcess())
+ if (IsAutoVacuumLauncherProcess() || IsHeapCleanerLauncherProcess())
{
/* report this backend in the PgBackendStatus array */
pgstat_bestart();
@@ -722,7 +723,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
* In standalone mode and in autovacuum worker processes, we use a fixed
* ID, otherwise we figure it out from the authenticated user name.
*/
- if (bootstrap || IsAutoVacuumWorkerProcess())
+ if (bootstrap || IsAutoVacuumWorkerProcess() || IsHeapCleanerWorkerProcess())
{
InitializeSessionUserIdStandalone();
am_superuser = true;
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 6a6b467fee..698b007d65 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,16 @@
#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+/* Progress parameters for cleaner */
+#define PROGRESS_CLEANER_RELATIONS 0
+#define PROGRESS_CLEANER_TOTAL_QUEUE_LENGTH 1
+#define PROGRESS_CLEANER_BUF_NINMEM 2
+#define PROGRESS_CLEANER_TOTAL_DELETIONS 3
+#define PROGRESS_CLEANER_VAINLY_CLEANED_TUPLES 4
+#define PROGRESS_CLEANER_MISSED_BLOCKS 5
+#define PROGRESS_CLEANER_CLEANED_BLOCKS 6
+#define PROGRESS_CLEANER_VAINLY_CLEANED_BLOCKS 7
+#define PROGRESS_CLEANER_NACQUIRED_LOCKS 8
+#define PROGRESS_CLEANER_TIMEOUT 9
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 85d472f0a5..705120eddc 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -186,6 +186,9 @@ extern void vacuum_set_xid_limits(Relation rel,
extern void vac_update_datfrozenxid(void);
extern void vacuum_delay_point(void);
+extern void quick_vacuum_index(Relation irel, Relation hrel,
+ ItemPointer dead_tuples,
+ int num_dead_tuples);
/* in commands/vacuumlazy.c */
extern void lazy_vacuum_rel(Relation onerel, int options,
VacuumParams *params, BufferAccessStrategy bstrategy);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d59c24ae23..dbf09328ac 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -56,6 +56,7 @@ typedef enum StatMsgType
PGSTAT_MTYPE_RESETSHAREDCOUNTER,
PGSTAT_MTYPE_RESETSINGLECOUNTER,
PGSTAT_MTYPE_AUTOVAC_START,
+ PGSTAT_MTYPE_BGHEAP_START,
PGSTAT_MTYPE_VACUUM,
PGSTAT_MTYPE_ANALYZE,
PGSTAT_MTYPE_ARCHIVER,
@@ -354,6 +355,12 @@ typedef struct PgStat_MsgAutovacStart
TimestampTz m_start_time;
} PgStat_MsgAutovacStart;
+typedef struct PgStat_MsgHeapCleanerStart
+{
+ PgStat_MsgHdr m_hdr;
+ Oid m_databaseid;
+ TimestampTz m_start_time;
+} PgStat_MsgHeapCleanerStart;
/* ----------
* PgStat_MsgVacuum Sent by the backend or autovacuum daemon
@@ -704,6 +711,8 @@ typedef enum BackendType
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
+ B_BG_HEAPCLNR_LAUNCHER,
+ B_BG_HEAPCLNR_WORKER,
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
@@ -756,6 +765,7 @@ typedef enum
{
WAIT_EVENT_ARCHIVER_MAIN = PG_WAIT_ACTIVITY,
WAIT_EVENT_AUTOVACUUM_MAIN,
+ WAIT_EVENT_BGHEAP_MAIN,
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
@@ -933,7 +943,8 @@ typedef enum
typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
- PROGRESS_COMMAND_VACUUM
+ PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_CLEANER
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 10
@@ -1181,6 +1192,7 @@ extern void pgstat_reset_shared_counters(const char *);
extern void pgstat_reset_single_counter(Oid objectid, PgStat_Single_Reset_Type type);
extern void pgstat_report_autovac(Oid dboid);
+extern void pgstat_report_heapcleaner(Oid dboid);
extern void pgstat_report_vacuum(Oid tableoid, bool shared,
PgStat_Counter livetuples, PgStat_Counter deadtuples);
extern void pgstat_report_analyze(Relation rel,
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5370035f0c..58be48198b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -333,6 +333,7 @@ extern void LocalPrefetchBuffer(SMgrRelation smgr, ForkNumber forkNum,
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
BlockNumber blockNum, bool *foundPtr);
extern void MarkLocalBufferDirty(Buffer buffer);
+extern bool IsLocalBufferDirty(Buffer buffer);
extern void DropRelFileNodeLocalBuffers(RelFileNode rnode, ForkNumber forkNum,
BlockNumber firstDelBlock);
extern void DropRelFileNodeAllLocalBuffers(RelFileNode rnode);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 3cce3906a0..07536fc7e7 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -43,8 +43,10 @@ typedef enum
RBM_ZERO_AND_CLEANUP_LOCK, /* Like RBM_ZERO_AND_LOCK, but locks the page
* in "cleanup" mode */
RBM_ZERO_ON_ERROR, /* Read, but return an all-zeros page on error */
- RBM_NORMAL_NO_LOG /* Don't log page as invalid during WAL
+ RBM_NORMAL_NO_LOG, /* Don't log page as invalid during WAL
* replay; otherwise same as RBM_NORMAL */
+ RBM_NORMAL_NO_READ /* Don't actually read page, returns only if it
+ * presents in shared memory */
} ReadBufferMode;
/* forward declared, to avoid having to expose buf_internals.h here */
@@ -175,6 +177,7 @@ extern Buffer ReadBufferWithoutRelcache(RelFileNode rnode,
extern void ReleaseBuffer(Buffer buffer);
extern void UnlockReleaseBuffer(Buffer buffer);
extern void MarkBufferDirty(Buffer buffer);
+extern bool IsBufferDirty(Buffer buffer);
extern void IncrBufferRefCount(Buffer buffer);
extern Buffer ReleaseAndReadBuffer(Buffer buffer, Relation relation,
BlockNumber blockNum);
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 5ecc1b757c..4c73acd62c 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -36,6 +36,8 @@ typedef enum
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
+ PMSIGNAL_START_HEAPCLNR_LAUNCHER,
+ PMSIGNAL_START_HEAPCLNR_WORKER,
PMSIGNAL_BACKGROUND_WORKER_CHANGE, /* background worker state change */
PMSIGNAL_START_WALRECEIVER, /* start a walreceiver */
PMSIGNAL_ADVANCE_STATE_MACHINE, /* advance postmaster's state machine */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..478e8e698c 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -167,7 +167,7 @@ select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
a | b
---+----
1 | 11
@@ -1821,6 +1821,22 @@ pg_stat_database_conflicts| SELECT d.oid AS datid,
pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin,
pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock
FROM pg_database d;
+pg_stat_progress_cleaner| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.relid,
+ s.param1 AS relations,
+ s.param2 AS queue_len,
+ s.param3 AS ninmem,
+ s.param4 AS cleaned_tuples,
+ s.param5 AS vain_tuples,
+ s.param6 AS missed_blocks,
+ s.param7 AS cleaned_blocks,
+ s.param8 AS vain_blocks,
+ s.param9 AS nlocks,
+ s.param10 AS timeout
+ FROM (pg_stat_get_progress_info('CLEANER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index e57c6e1c42..b9c0af7bce 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -637,7 +637,7 @@ select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
i
---
3
diff --git a/src/test/regress/input/constraints.source b/src/test/regress/input/constraints.source
index 98dd4210e9..2968a1230b 100644
--- a/src/test/regress/input/constraints.source
+++ b/src/test/regress/input/constraints.source
@@ -337,7 +337,7 @@ ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- explicitly defer the constraint
BEGIN;
@@ -349,7 +349,7 @@ DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- try adding an initially deferred constraint
ALTER TABLE unique_tbl DROP CONSTRAINT unique_tbl_i_key;
@@ -367,7 +367,7 @@ DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- should fail at commit-time
BEGIN;
@@ -420,7 +420,7 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- test a HOT update that modifies the newly inserted tuple,
-- but should succeed because we then remove the other conflicting tuple.
@@ -431,11 +431,11 @@ INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
DROP TABLE unique_tbl;
diff --git a/src/test/regress/output/constraints.source b/src/test/regress/output/constraints.source
index a6a1df18e7..a2ae6df587 100644
--- a/src/test/regress/output/constraints.source
+++ b/src/test/regress/output/constraints.source
@@ -478,7 +478,7 @@ DETAIL: Key (i)=(1) already exists.
ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+------
1 | one
@@ -494,14 +494,14 @@ SET CONSTRAINTS unique_tbl_i_key DEFERRED;
INSERT INTO unique_tbl VALUES (3, 'three');
DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
1 | one
2 | two
+ 3 | three
4 | four
5 | five
- 3 | three
(5 rows)
-- try adding an initially deferred constraint
@@ -516,14 +516,14 @@ UPDATE unique_tbl SET i = 2 WHERE i = 4 AND t = 'four';
DELETE FROM unique_tbl WHERE i = 1 AND t = 'one';
DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- should fail at commit-time
@@ -581,14 +581,14 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
ERROR: duplicate key value violates unique constraint "unique_tbl_i_key"
DETAIL: Key (i)=(3) already exists.
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- test a HOT update that modifies the newly inserted tuple,
@@ -597,25 +597,25 @@ BEGIN;
INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
DROP TABLE unique_tbl;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 3ca4c07356..c540f12a29 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -181,7 +181,7 @@ delete from rtest_v1 where b = 12;
select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
** Remember the delete rule on rtest_v1: It says
** DO INSTEAD DELETE FROM rtest_t1 WHERE a = old.a
** So this time both rows with a = 2 must get deleted
diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql
index 1250cd63e0..5a45fbad41 100644
--- a/src/test/regress/sql/triggers.sql
+++ b/src/test/regress/sql/triggers.sql
@@ -419,7 +419,7 @@ delete from trigtest where i=1;
select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
drop table trigtest2;
drop table trigtest;
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-09-05 12:25 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-09-06 10:26 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
0 siblings, 0 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-09-06 10:26 UTC (permalink / raw)
To: pgsql-hackers
And background worker patch in attachment.
05.09.2018 15:25, Andrey Lepikhov пишет:
> Hi,
>
> I prepared next version of Background worker (cleaner) based on a retail
> indextuple deletion patch.
> This version shows stable behavior on regression tests and pgbench
> workloads.
>
> In this version:
> 1. Only AccessShareLock are acquired on a cleanup of heap and index
> relations.
> 2. Some 'aggressive' cleanup strategy introduced - conditional cleanup
> locks not used.
> 3. Cleanup only an in-memory blocks.
> 4. The Cleaner calls heap_page_prune() before cleanup a block.
>
> Benchmarks
> ---------
>
> Two factors were evaluated: performance (tps) and relations blowing.
>
> Before each test some rarefaction of pgbench_accounts was modeled by
> deletion 10% of tuples at each block.
> Also, I tested normal and Gaussian distribution of queries on
> pgbench_accounts relation.
> Autovacuum uses default settings.
>
> Script:
> pgbench -i -s 10
> psql -c $"DELETE FROM pgbench_accounts WHERE (random() < 0.1);"
> psql -c $"VACUUM;"
> psql -c $"CREATE INDEX pgbench_accounts_ext ON public.pgbench_accounts
> USING btree (abalance);" &&
> pgbench -T 3600 -c 32 -j 8 -M prepared -P 600
>
> NORMAL distribution:
> average tps = 1045 (cleaner); = 1077 (autovacuum)
>
> Relations size at the end of test, MB:
> pgbench_accounts: 128 (cleaner); 128 (autovacuum)
> pgbench_branches: 0.1 (cleaner); 2.1 (autovacuum)
> pgbench_tellers: 0.4 (cleaner); 2.8 (autovacuum)
> pgbench_accounts_pkey: 21 (cleaner); 43 (autovacuum)
> pgbench_accounts_ext: 48 (cleaner); 56 (autovacuum)
>
> Gaussian distribution:
> average tps = 213 (cleaner); = 213 (autovacuum)
>
> Relations size at the end of test, MB:
> pgbench_accounts: 128 (cleaner); 128 (autovacuum)
> pgbench_accounts_ext: 22 (cleaner); 29 (autovacuum)
>
> Conclusions
> -----------
> 1. For retail indextuple deletion purposes i replaced ItemIdSetDead() by
> ItemIdMarkDead() in heap_page_prune_execute() operation. Hereupon in the
> case of 100% filling of each relation block we get a blowing HEAP and
> index , more or less. When the blocks already have free space, the
> cleaner can delay blowing the heap and index without a vacuum.
> 2. Cleaner works fine in the case of skewness of access frequency to
> relation blocks.
> 3. The cleaner does not cause a decrease of performance.
>
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments:
0005-Heap-and-Index-cleaner.patch (text/x-patch, 113.8K) download
From b092dfd95b8673d692730ac27a1d6fdb76b66601 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Thu, 6 Sep 2018 11:05:42 +0300
Subject: [PATCH 5/5] Heap-and-Index-cleaner
---
.../postgres_fdw/expected/postgres_fdw.out | 30 +-
contrib/postgres_fdw/sql/postgres_fdw.sql | 4 +-
src/backend/access/heap/pruneheap.c | 6 +-
src/backend/access/nbtree/nbtree.c | 15 +-
src/backend/access/transam/xact.c | 4 +
src/backend/catalog/system_views.sql | 11 +
src/backend/commands/vacuumlazy.c | 44 +-
src/backend/postmaster/Makefile | 2 +-
src/backend/postmaster/bgheap.c | 1990 +++++++++++++++++
src/backend/postmaster/pgstat.c | 36 +
src/backend/postmaster/postmaster.c | 160 +-
src/backend/storage/buffer/bufmgr.c | 60 +-
src/backend/storage/buffer/localbuf.c | 24 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/storage/lmgr/proc.c | 5 +-
src/backend/tcop/postgres.c | 12 +
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/backend/utils/hash/Makefile | 2 +-
src/backend/utils/hash/shash.c | 265 +++
src/backend/utils/init/miscinit.c | 3 +-
src/backend/utils/init/postinit.c | 11 +-
src/include/commands/progress.h | 12 +
src/include/commands/vacuum.h | 3 +
src/include/pgstat.h | 14 +-
src/include/postmaster/bgheap.h | 36 +
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 5 +-
src/include/storage/pmsignal.h | 2 +
src/include/utils/shash.h | 54 +
src/test/regress/expected/rules.out | 18 +-
src/test/regress/expected/triggers.out | 2 +-
src/test/regress/input/constraints.source | 12 +-
src/test/regress/output/constraints.source | 34 +-
src/test/regress/sql/rules.sql | 2 +-
src/test/regress/sql/triggers.sql | 2 +-
36 files changed, 2803 insertions(+), 84 deletions(-)
create mode 100644 src/backend/postmaster/bgheap.c
create mode 100644 src/backend/utils/hash/shash.c
create mode 100644 src/include/postmaster/bgheap.h
create mode 100644 src/include/utils/shash.h
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index f5498c62bd..622bea2a7d 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -5496,19 +5496,25 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
- QUERY PLAN
-----------------------------------------------------------------------------------------------------------
- Update on public.ft2
- Output: c1, c2, c3, c4, c5, c6, c7, c8
- Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
- -> Foreign Scan on public.ft2
- Output: c1, c2, NULL::integer, 'bar'::text, c4, c5, c6, c7, c8, ctid
- Filter: (postgres_fdw_abs(ft2.c1) > 2000)
- Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
-(7 rows)
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------
+ Sort
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+ Sort Key: updated.c1
+ CTE updated
+ -> Update on public.ft2
+ Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: ft2.c1, ft2.c2, NULL::integer, 'bar'::text, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.ctid
+ Filter: (postgres_fdw_abs(ft2.c1) > 2000)
+ Remote SQL: SELECT "C 1", c2, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" FOR UPDATE
+ -> CTE Scan on updated
+ Output: updated.c1, updated.c2, updated.c3, updated.c4, updated.c5, updated.c6, updated.c7, updated.c8
+(13 rows)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
------+----+-----+----+----+----+------------+----
2001 | 1 | bar | | | | ft2 |
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index e1b955f3f0..6034e83108 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1137,8 +1137,8 @@ ALTER SERVER loopback OPTIONS (DROP extensions);
INSERT INTO ft2 (c1,c2,c3)
SELECT id, id % 10, to_char(id, 'FM00000') FROM generate_series(2001, 2010) id;
EXPLAIN (verbose, costs off)
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *; -- can't be pushed down
-UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING *;
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1; -- can't be pushed down
+with updated AS ( UPDATE ft2 SET c3 = 'bar' WHERE postgres_fdw_abs(c1) > 2000 RETURNING * ) SELECT * FROM updated ORDER BY c1;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c3 = 'baz'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index ebbafe3275..e7c2d897f3 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -22,6 +22,7 @@
#include "catalog/catalog.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
@@ -156,6 +157,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/* OK to prune */
(void) heap_page_prune(relation, buffer, OldestXmin, true, &ignore);
+ if (IsBufferDirty(buffer))
+ HeapCleanerSend(relation, BufferGetBlockNumber(buffer));
}
/* And release buffer lock */
@@ -411,7 +414,6 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
- ndeleted++;
}
/* Nothing more to do */
@@ -583,9 +585,9 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- ndeleted++;
if (chainitems[i] == latestdead)
continue;
+ ndeleted++;
heap_prune_record_unused(prstate, chainitems[i]);
}
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 50378e0f51..28650008ef 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -930,7 +930,14 @@ bttargetdelete(IndexTargetDeleteInfo *info,
skey = _bt_mkscankey(irel, itup);
/* Descend the tree and position ourselves on the target leaf page. */
- stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_READ, NULL);
+ stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_WRITE, NULL);
+
+ /* trade in our read lock for a write lock */
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBuffer(buf, BT_WRITE);
+
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_WRITE, NULL);
/* To prepare tuple entries search across index pages */
Assert(BufferIsValid(buf));
@@ -954,6 +961,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
@@ -974,7 +982,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
* Traverse to a next reliable index page
*/
buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
- false, true, stack, BT_READ, NULL);
+ false, true, stack, BT_WRITE, NULL);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
@@ -1019,6 +1027,7 @@ bttargetdelete(IndexTargetDeleteInfo *info,
}
else if (res > 0)
break;
+
pos--;
}
@@ -1033,11 +1042,11 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
if (ndeletable > 0)
{
+ /* trade in our read lock for a write lock */
LockBuffer(buf, BUFFER_LOCK_UNLOCK);
LockBufferForCleanup(buf);
_bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
-
stats->tuples_removed += ndeletable;
}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9aa63c8792..f5e70ae16d 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -40,6 +40,7 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgheap.h"
#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
@@ -2126,6 +2127,7 @@ CommitTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(true);
AtEOXact_Snapshot(true, false);
AtEOXact_ApplyLauncher(true);
@@ -2404,6 +2406,7 @@ PrepareTransaction(void)
AtEOXact_Files(true);
AtEOXact_ComboCid();
AtEOXact_HashTables(true);
+ AtEOXact_BGHeap_tables(true);
/* don't call AtEOXact_PgStat here; we fixed pgstat state above */
AtEOXact_Snapshot(true, true);
pgstat_report_xact_timestamp(0);
@@ -2606,6 +2609,7 @@ AbortTransaction(void)
AtEOXact_Files(false);
AtEOXact_ComboCid();
AtEOXact_HashTables(false);
+ AtEOXact_BGHeap_tables(true);
AtEOXact_PgStat(false);
AtEOXact_ApplyLauncher(false);
pgstat_report_xact_timestamp(0);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 7251552419..3b6e9aa41c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -903,6 +903,17 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_cleaner AS
+ SELECT
+ S.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid,
+ S.param1 AS relations, S.param2 AS queue_len, S.param3 AS ninmem,
+ S.param4 AS cleaned_tuples, S.param5 AS vain_tuples,
+ S.param6 AS missed_blocks, S.param7 AS cleaned_blocks,
+ S.param8 AS vain_blocks, S.param9 AS nlocks, S.param10 AS timeout
+ FROM pg_stat_get_progress_info('CLEANER') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid;
+
CREATE VIEW pg_user_mappings AS
SELECT
U.oid AS umid,
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 1b952fedd9..9b101e79fb 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -158,8 +158,6 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
-static void quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -742,7 +740,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1394,7 +1392,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
- quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ quick_vacuum_index(Irel[i], onerel, vacrelstats->dead_tuples, vacrelstats->num_dead_tuples);
else
lazy_vacuum_index(Irel[i],
&indstats[i],
@@ -1730,7 +1728,12 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
{
offnum = ItemIdGetRedirect(lp);
lp = PageGetItemId(page, offnum);
- Assert(ItemIdIsUsed(lp));
+
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
}
/* Form a tuple */
@@ -1747,30 +1750,34 @@ get_tuple_by_tid(Relation rel, ItemPointer tid)
* quick_vacuum_index() -- quick vacuum one index relation.
*
* Delete all the index entries pointing to tuples listed in
- * vacrelstats->dead_tuples.
+ * dead_tuples.
*/
-static void
+void
quick_vacuum_index(Relation irel, Relation hrel,
- LVRelStats *vacrelstats)
+ ItemPointer dead_tuples,
+ int num_dead_tuples)
{
int tnum;
- bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
- IndexInfo* indexInfo = BuildIndexInfo(irel);
- EState* estate = CreateExecutorState();
- ExprContext* econtext = GetPerTupleExprContext(estate);
- ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
- TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ bool *found = palloc0(num_dead_tuples*sizeof(bool));
+ IndexInfo *indexInfo = BuildIndexInfo(irel);
+ EState *estate = CreateExecutorState();
+ ExprContext *econtext = GetPerTupleExprContext(estate);
+ ExprState *predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
IndexTargetDeleteResult stats;
IndexTargetDeleteInfo ivinfo;
+ Assert(found != NULL);
+
+ stats.tuples_removed = 0;
ivinfo.indexRelation = irel;
ivinfo.heapRelation = hrel;
econtext->ecxt_scantuple = slot;
/* Get tuple from heap */
- for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
+ for (tnum = num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1781,7 +1788,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
continue;
/* Get a tuple from heap */
- if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ if ((tuple = get_tuple_by_tid(hrel, &(dead_tuples[tnum]))) == NULL)
{
/*
* Tuple has 'not used' status.
@@ -1814,15 +1821,16 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.dead_tuples = dead_tuples;
ivinfo.last_dead_tuple = tnum;
ivinfo.found_dead_tuples = found;
+
index_target_delete(&ivinfo, &stats, values, isnull);
}
- pfree(found);
ExecDropSingleTupleTableSlot(slot);
FreeExecutorState(estate);
+ pfree(found);
}
/*
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 71c23211b2..11796375e8 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -12,7 +12,7 @@ subdir = src/backend/postmaster
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = autovacuum.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
+OBJS = autovacuum.o bgheap.o bgworker.o bgwriter.o checkpointer.o fork_process.o \
pgarch.o pgstat.o postmaster.o startup.o syslogger.o walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/bgheap.c b/src/backend/postmaster/bgheap.c
new file mode 100644
index 0000000000..c445b303b3
--- /dev/null
+++ b/src/backend/postmaster/bgheap.c
@@ -0,0 +1,1990 @@
+/*
+ * bgheap.c
+ *
+ * PostgreSQL integrated cleaner of HEAP and INDEX relations
+ *
+ * Made in autovacuum analogy. Uses 'Target' strategy for clean relations,
+ * without full scan.
+ * The cleaner to consist of one Launcher and Workers.
+ * One worker corresponds to one database.
+ * Launcher receives message from a backend {dbOid; relOid; blkno} by
+ * socket and translate it to a worker by shared memory buffer. Worker receive
+ * message and cleanup block of heap relation and its index relations.
+ *
+ * Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/bgheap.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <unistd.h>
+#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+#include "access/genam.h"
+#include "access/heapam.h"
+#include "access/heapam_xlog.h"
+#include "access/htup_details.h"
+#include "access/nbtree.h"
+#include "access/visibilitymap.h"
+#include "access/xact.h"
+#include "catalog/catalog.h"
+#include "catalog/pg_database.h"
+#include "catalog/pg_namespace.h"
+#include "catalog/index.h"
+#include "catalog/namespace.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
+#include "common/ip.h"
+#include "executor/executor.h"
+#include "libpq/pqsignal.h"
+#include "postmaster/bgheap.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/postmaster.h"
+#include "storage/condition_variable.h"
+#include "storage/fd.h"
+#include "storage/freespace.h"
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/pmsignal.h"
+#include "storage/procarray.h"
+#include "storage/smgr.h"
+#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
+#include "utils/lsyscache.h"
+#include "utils/resowner.h"
+#include "utils/shash.h"
+#include "utils/syscache.h"
+#include "utils/timeout.h"
+
+/*
+ * Maximum number of task items in storage at a backend side before shipping to a
+ * background heap cleaner
+ */
+#define BACKEND_DIRTY_ITEMS_MAX (10)
+
+/*
+ * Maximum number of task items in waiting list at a Launcher side.
+ * It is necessary to make it sufficiently large. If a number of arrived
+ * messages exceed this, we ignore the remains and increase counter of missed
+ * items.
+ */
+#define WAITING_MESSAGES_MAX_NUM (100000)
+
+/*
+ * Maximum number of dirty blocks which can keep a worker for each relation
+ */
+#define WORKER_DIRTYBLOCKS_MAX_NUM (100000)
+
+/* Maximum number of slots for dirty relations */
+#define WORKER_RELATIONS_MAX_NUM (100)
+
+/* Maximum number of task items in a launcher/worker shared buffer */
+#define WORKER_TASK_ITEMS_MAX (1000)
+
+/*
+ * Maximum time interval which worker can idle without a task (ms)
+ */
+#define WORKER_IDLE_TIME_DURATION_MAX (5000)
+
+/* Minimal info for cleanup a block of heap relation and its index relations */
+typedef struct CleanerMessage
+{
+ Oid dbNode; /* Database ID */
+ Oid relid; /* Relation ID */
+ int blkno; /* Block number */
+ TransactionId xid;
+ uint16 hits;
+} CleanerMessage;
+
+typedef struct WorkerTask
+{
+ BlockNumber blkno;
+ TransactionId lastXid;
+ uint16 hits;
+} WorkerTask;
+
+/*
+ * Shared memory data to control and two-way communication with worker
+ */
+typedef struct WorkerInfoData
+{
+ dlist_node links;
+ Oid dbOid; /* Database ID of the worker */
+ TimestampTz launchtime; /* To define a time of last worker activity */
+ int pid; /* Used for system signals passing */
+ CleanerMessage buffer[WORKER_TASK_ITEMS_MAX]; /* Array of work items */
+ int nitems; /* Number of work items in buffer */
+ LWLock WorkItemLock; /* Locker for safe buffer access */
+ int id; /* Used for Internal launcher buffers management */
+} WorkerInfoData;
+
+typedef struct WorkerInfoData *WorkerInfo;
+
+/*
+ * Shared memory info about workers pool.
+ * Launcher get an element from freeWorkers list and init startingWorker value.
+ * Worker set startingWorker to NULL value after startup and add himself
+ * to runningWorkers list.
+ */
+typedef struct HeapCleanerShmemStruct
+{
+ dlist_head freeWorkers;
+ dlist_head runningWorkers;
+ WorkerInfo startingWorker;
+} HeapCleanerShmemStruct;
+
+typedef struct DirtyRelation
+{
+ Oid relid;
+ SHTAB *items;
+} DirtyRelation;
+
+static int stat_vainly_cleaned_tuples = 0;
+static int stat_vainly_cleaned_blocks = 0;
+static uint64 stat_total_deletions = 0;
+static uint64 stat_total_cleaned_blocks = 0;
+static uint64 stat_not_acquired_locks = 0;
+static int stat_buf_ninmem = 0;
+static uint32 stat_missed_blocks = 0;
+
+static MemoryContext BGHeapMemCxt = NULL;
+
+/*
+ * Table of database relations.
+ * For each relation we support a waiting list of dirty blocks.
+ */
+static PSHTAB PrivateRelationsTable = NULL;
+
+static bool am_heapcleaner_launcher = false;
+static bool am_heapcleaner_worker = false;
+
+int heapcleaner_max_workers = 10;
+static WorkerInfo MyWorkerInfo = NULL;
+static HeapCleanerShmemStruct *HeapCleanerShmem;
+
+/*
+ * Hash table for collection of dirty blocks after heap_page_prune() action
+ * at a Backend side.
+ */
+PSHTAB dblocks = NULL;
+
+/* Signal handling */
+static volatile sig_atomic_t got_SIGHUP = false;
+static volatile sig_atomic_t got_SIGTERM = false;
+static volatile sig_atomic_t got_SIGUSR2 = false;
+
+NON_EXEC_STATIC pgsocket HeapCleanerSock = PGINVALID_SOCKET;
+static struct sockaddr_storage HeapCleanerSockAddr;
+
+static int TrancheId;
+
+/*
+ * Hash table for waiting data: one for each active worker and one for all messages
+ * intended to not-running workers. Reasons:
+ * 1. Worker in startup process.
+ * 2. Another worker in startup and we can't launch new worker.
+ * 3. We have not free slots for new workers.
+ */
+PSHTAB *wTab;
+
+/*
+ * Parameters of each hash table
+ */
+SHTABCTL wTabCtl;
+
+#define CLEANUP_GENTLY (0)
+#define CLEANUP_AGGRESSIVE (1)
+
+/*
+ * Cleanup strategy
+ */
+static int strategy = CLEANUP_AGGRESSIVE;
+
+#ifdef EXEC_BACKEND
+static pid_t hclauncher_forkexec(void);
+static pid_t hcworker_forkexec(void);
+#endif
+
+static uint64 CleanerMessageHashFunc(void *key, uint64 size, uint64 base);
+static PSHTAB cleanup_relations(DirtyRelation *res, SHTAB *AuxiliaryList, bool got_SIGTERM);
+static bool DefaultCompareFunc(void* bucket1, void* bucket2);
+NON_EXEC_STATIC void HeapCleanerLauncherMain(int argc, char *argv[]) pg_attribute_noreturn();
+NON_EXEC_STATIC void HeapCleanerWorkerMain(int argc, char *argv[]) pg_attribute_noreturn();
+static bool isEqualMsgs(void *arg1, void *arg2);
+
+static void launch_worker(Oid dbNode);
+static WorkerInfo look_for_worker(Oid dbNode);
+static void main_launcher_loop(void);
+static void main_worker_loop(void);
+
+static void SIGHUP_Handler(SIGNAL_ARGS);
+static void SIGTERM_Handler(SIGNAL_ARGS);
+static void SIGUSR2_Handler(SIGNAL_ARGS);
+static void backend_send_dirty_blocks(void);
+
+#ifdef EXEC_BACKEND
+/*
+ * forkexec routine for the autovacuum launcher process.
+ *
+ * Format up the arglist, then fork and exec.
+ */
+static pid_t
+hclauncher_forkexec(void)
+{
+ char *av[10];
+ int ac = 0;
+
+ av[ac++] = "postgres";
+ av[ac++] = "--forkhclauncher";
+ av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ av[ac] = NULL;
+
+ Assert(ac < lengthof(av));
+
+ return postmaster_forkexec(ac, av);
+}
+
+/*
+ * We need this set from the outside, before InitProcess is called
+ */
+void
+HeapCleanerLauncherIAm(void)
+{
+ am_heapcleaner_launcher = true;
+}
+
+static pid_t
+hcworker_forkexec(void)
+{
+ char *av[10];
+ int ac = 0;
+
+ av[ac++] = "postgres";
+ av[ac++] = "--forkhcworker";
+ av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ av[ac] = NULL;
+
+ Assert(ac < lengthof(av));
+
+ return postmaster_forkexec(ac, av);
+}
+
+/*
+ * We need this set from the outside, before InitProcess is called
+ */
+void
+HeapCleanerWorkerIAm(void)
+{
+ am_heapcleaner_worker = true;
+}
+#endif
+
+/*
+ * Send dirty blocks from a collector's table to launcher at the end of transaction
+ */
+void
+AtEOXact_BGHeap_tables(bool isCommit)
+{
+ MemoryContext oldMemCxt;
+
+ if (BGHeapMemCxt == NULL)
+ BGHeapMemCxt = AllocSetContextCreate(BGHeapMemCxt, "bgheap",
+ ALLOCSET_DEFAULT_SIZES);
+ oldMemCxt = MemoryContextSwitchTo(BGHeapMemCxt);
+
+ backend_send_dirty_blocks();
+
+ MemoryContextSwitchTo(oldMemCxt);
+}
+
+static void
+save_to_list(PSHTAB AuxiliaryList, WorkerTask *item)
+{
+ bool found;
+
+ WorkerTask *new_item = (WorkerTask *)
+ SHASH_Search(AuxiliaryList,
+ (void *) &(item->blkno),
+ HASH_ENTER, &found);
+
+ Assert((new_item != NULL) && (!found));
+ new_item->hits = item->hits;
+ new_item->lastXid = item->lastXid;
+}
+
+/*
+ * Main logic of HEAP and index relations cleaning
+ */
+static PSHTAB
+cleanup_relations(DirtyRelation *res, PSHTAB AuxiliaryList, bool got_SIGTERM)
+{
+ Relation heapRelation;
+ Relation *IndexRelations;
+ int nindexes;
+ LOCKMODE lockmode = AccessShareLock;
+ WorkerTask *item;
+ TransactionId OldestXmin;
+
+ Assert(res != NULL);
+ Assert(res->items != NULL);
+ Assert(AuxiliaryList != NULL);
+ Assert(SHASH_Entries(AuxiliaryList) == 0);
+
+ if (SHASH_Entries(res->items) == 0)
+ return AuxiliaryList;
+
+ if (RecoveryInProgress())
+ {
+ SHASH_Clean(res->items);
+ return AuxiliaryList;
+ }
+
+ CHECK_FOR_INTERRUPTS();
+
+ StartTransactionCommand();
+
+ /* Setting a snapshot ensures that RecentGlobalXmin is kept truly recent. */
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /*
+ * At this point relation availability is not guaranteed.
+ * Make safe test to check this.
+ */
+ heapRelation = try_relation_open(res->relid, lockmode);
+
+ if (!heapRelation)
+ {
+ elog(LOG, "[%d] Cleanup: UnSuccessful opening. Relation deleted? dbOid=%d", res->relid, MyWorkerInfo->dbOid);
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+ SHASH_Clean(res->items);
+ return AuxiliaryList;
+ }
+
+ /*
+ * Check relation type similarly vacuum
+ */
+ if (!(pg_class_ownercheck(RelationGetRelid(heapRelation), GetUserId()) ||
+ (pg_database_ownercheck(MyDatabaseId, GetUserId()) && !heapRelation->rd_rel->relisshared)))
+ {
+ if (heapRelation->rd_rel->relisshared)
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- only superuser can vacuum it",
+ RelationGetRelationName(heapRelation))));
+ else if (heapRelation->rd_rel->relnamespace == PG_CATALOG_NAMESPACE)
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- only superuser or database owner can vacuum it",
+ RelationGetRelationName(heapRelation))));
+ else
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- only table or database owner can vacuum it",
+ RelationGetRelationName(heapRelation))));
+ relation_close(heapRelation, lockmode);
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+ SHASH_Clean(res->items);
+ return AuxiliaryList;
+ }
+
+ /*
+ * Get xid for following prune process. Function here is rather
+ * expensive, so call them only one per round.
+ */
+ OldestXmin = GetOldestXmin(heapRelation, PROCARRAY_FLAGS_VACUUM);
+
+ Assert(TransactionIdIsValid(OldestXmin));
+ /* Main cleanup cycle */
+ for (SHASH_SeqReset(res->items);
+ (item = (WorkerTask *) SHASH_SeqNext(res->items)) != NULL; )
+ {
+ BlockNumber nblocks;
+ Buffer buffer;
+ ItemPointerData dead_tuples[MaxOffsetNumber];
+ int num_dead_tuples = 0;
+ OffsetNumber offnum;
+ int irnum;
+ Page page;
+ ItemId lp;
+ int tnum;
+ bool found_non_nbtree = false;
+ TransactionId latestRemovedXid;
+
+ Assert(item->hits > 0);
+
+ nblocks = RelationGetNumberOfBlocks(heapRelation);
+
+ if (item->blkno >= nblocks)
+ /*
+ * Block was deleted early.
+ * Skip cleaning and drop block from waiting list.
+ */
+ continue;
+
+ /*
+ * Get and pin the buffer.
+ * If Postgres not in termination state when we get in-memory buffer only
+ */
+ if (!got_SIGTERM)
+ buffer = ReadBufferExtended(heapRelation, MAIN_FORKNUM, item->blkno, RBM_NORMAL_NO_READ, NULL);
+ else
+ buffer = ReadBuffer(heapRelation, item->blkno);
+
+ if (BufferIsInvalid(buffer))
+ {
+ /*
+ * Buffer was evicted from shared buffers already.
+ */
+ Assert(!got_SIGTERM);
+ stat_buf_ninmem++;
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+
+ /*
+ * stop cleaning if page was changed by transaction after OldestXmin. In
+ * this case there is high probability that we can't do anything usefull
+ * with it. Let we return to clean later.
+ */
+ if (!got_SIGTERM && !IsBufferDirty(buffer) && (!TransactionIdPrecedesOrEquals(item->lastXid, OldestXmin)))
+ {
+ ReleaseBuffer(buffer);
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+
+ /*
+ * Lock the buffer for pruning
+ */
+ if (strategy == CLEANUP_GENTLY)
+ {
+ if (!ConditionalLockBufferForCleanup(buffer))
+ {
+ stat_not_acquired_locks++;
+ pgstat_progress_update_param(PROGRESS_CLEANER_NACQUIRED_LOCKS, stat_not_acquired_locks);
+
+ /* Can't lock buffer. */
+ ReleaseBuffer(buffer);
+ save_to_list(AuxiliaryList, item);
+ continue;
+ }
+ }
+ else if (strategy == CLEANUP_AGGRESSIVE)
+ LockBufferForCleanup(buffer);
+
+ /*
+ * Increase our chances for cleaning more tuples.
+ */
+ (void) heap_page_prune(heapRelation, buffer, OldestXmin, false, &latestRemovedXid);
+
+ page = BufferGetPage(buffer);
+
+ /* Collect dead tuples TID's */
+ for (offnum = FirstOffsetNumber;
+ offnum < PageGetMaxOffsetNumber(page);
+ offnum = OffsetNumberNext(offnum))
+ {
+ lp = PageGetItemId(page, offnum);
+ if (ItemIdIsDead(lp))
+ {
+ ItemPointerSet(&(dead_tuples[num_dead_tuples]), item->blkno, offnum);
+ num_dead_tuples++;
+ }
+ }
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+
+ if (num_dead_tuples == 0)
+ {
+ /*
+ * Block is clear. Accumulate some stats and continue.
+ */
+ stat_vainly_cleaned_tuples += item->hits;
+ pgstat_progress_update_param(PROGRESS_CLEANER_VAINLY_CLEANED_TUPLES, stat_vainly_cleaned_tuples);
+ stat_vainly_cleaned_blocks++;
+ pgstat_progress_update_param(PROGRESS_CLEANER_VAINLY_CLEANED_BLOCKS, stat_vainly_cleaned_blocks);
+ ReleaseBuffer(buffer);
+ continue;
+ }
+
+ /* Open and lock index relations correspond to the heap relation */
+ vac_open_indexes(heapRelation, ShareLock, &nindexes, &IndexRelations);
+
+ /* Iterate across all index relations */
+ for (irnum = 0; irnum < nindexes; irnum++)
+ {
+ if (IndexRelations[irnum]->rd_amroutine->amtargetdelete == NULL)
+ {
+ /*
+ * Can clean only btree indexes now.
+ */
+ found_non_nbtree = true;
+ continue;
+ }
+
+ quick_vacuum_index(IndexRelations[irnum],
+ heapRelation,
+ dead_tuples,
+ num_dead_tuples);
+ }
+
+ vac_close_indexes(nindexes, IndexRelations, ShareLock);
+
+ /*
+ * If heap relation has not only b-tree indexes, can't clean heap block.
+ */
+ if (!found_non_nbtree)
+ {
+ OffsetNumber unusable[MaxOffsetNumber];
+ int nunusable = 0;
+ Size freespace;
+
+ if (strategy == CLEANUP_GENTLY)
+ {
+ if (!ConditionalLockBufferForCleanup(buffer))
+ {
+ ReleaseBuffer(buffer);
+ save_to_list(AuxiliaryList, item);
+ stat_not_acquired_locks++;
+ pgstat_progress_update_param(PROGRESS_CLEANER_NACQUIRED_LOCKS, stat_not_acquired_locks);
+ continue;
+ }
+ }
+ else if (strategy == CLEANUP_AGGRESSIVE)
+ LockBufferForCleanup(buffer);
+
+ START_CRIT_SECTION();
+
+ /* Release DEAD heap tuples storage */
+ for (tnum = 0; tnum < num_dead_tuples; tnum++)
+ {
+ OffsetNumber offnum = ItemPointerGetOffsetNumber(&dead_tuples[tnum]);
+ ItemId lp = PageGetItemId(page, offnum);
+
+ Assert(ItemIdIsDead(lp));
+ ItemIdSetUnused(lp);
+ unusable[nunusable++] = offnum;
+ }
+
+ if (nunusable > 0)
+ {
+ PageRepairFragmentation(page);
+ PageClearFull(page);
+ MarkBufferDirty(buffer);
+
+ if (RelationNeedsWAL(heapRelation))
+ {
+ XLogRecPtr recptr = log_heap_clean(heapRelation, buffer,
+ NULL, 0,
+ NULL, 0,
+ unusable, nunusable,
+ InvalidTransactionId);
+
+ PageSetLSN(BufferGetPage(buffer), recptr);
+ }
+ stat_total_deletions += nunusable;
+ stat_total_cleaned_blocks++;
+ pgstat_progress_update_param(PROGRESS_CLEANER_TOTAL_DELETIONS, stat_total_deletions);
+ pgstat_progress_update_param(PROGRESS_CLEANER_CLEANED_BLOCKS, stat_total_cleaned_blocks);
+ }
+
+ END_CRIT_SECTION();
+ freespace = PageGetHeapFreeSpace(page);
+
+ UnlockReleaseBuffer(buffer);
+ RecordPageWithFreeSpace(heapRelation, item->blkno, freespace);
+ pgstat_update_heap_dead_tuples(heapRelation, nunusable);
+ }
+ /*
+ * ToDo: that we will do with TOAST relation?
+ */
+ }
+
+ relation_close(heapRelation, lockmode);
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+
+ {
+ SHTAB *tmp = res->items;
+ SHASH_Clean(tmp);
+ res->items = AuxiliaryList;
+ return tmp;
+ }
+}
+
+static bool
+DefaultCompareFunc(void* bucket1, void* bucket2)
+{
+ WorkerTask *item1 = (WorkerTask *) bucket1;
+ WorkerTask *item2 = (WorkerTask *) bucket2;
+
+ if (item1->blkno == item2->blkno)
+ return true;
+ else
+ return false;
+}
+
+/*
+ * Free any launcher resources before close process
+ */
+static void
+FreeLauncherInfo(int code, Datum arg)
+{
+ int counter;
+
+ for (counter = 0; counter < heapcleaner_max_workers+1; counter++)
+ SHASH_Destroy(wTab[counter]);
+ pfree(wTab);
+}
+
+/*
+ * Return a WorkerInfo to the free list
+ */
+static void
+FreeWorkerInfo(int code, Datum arg)
+{
+ if (MyWorkerInfo != NULL)
+ {
+ LWLockAcquire(HeapCleanerLock, LW_EXCLUSIVE);
+
+ dlist_delete(&MyWorkerInfo->links);
+ MyWorkerInfo->dbOid = InvalidOid;
+ MyWorkerInfo->launchtime = 0;
+ dlist_push_head(&HeapCleanerShmem->freeWorkers,
+ &MyWorkerInfo->links);
+ /* not mine anymore */
+ MyWorkerInfo = NULL;
+
+ LWLockRelease(HeapCleanerLock);
+ }
+ else
+ elog(ERROR, "---> MyWorkerInfo is NULL");
+}
+
+/*
+ * Initialize communication with backends
+ */
+void
+HeapCleanerInit()
+{
+ struct addrinfo *addrs = NULL,
+ *addr, hints;
+ ACCEPT_TYPE_ARG3 alen;
+ int ret;
+ fd_set rset;
+ struct timeval tv;
+ char test_byte;
+ int sel_res;
+ int tries = 0;
+
+#define TESTBYTEVAL ((char) 199)
+
+ /*
+ * Create the UDP socket for sending and receiving statistic messages
+ */
+ hints.ai_flags = AI_PASSIVE;
+ hints.ai_family = AF_UNSPEC;
+ hints.ai_socktype = SOCK_DGRAM;
+ hints.ai_protocol = 0;
+ hints.ai_addrlen = 0;
+ hints.ai_addr = NULL;
+ hints.ai_canonname = NULL;
+ hints.ai_next = NULL;
+ ret = pg_getaddrinfo_all("localhost", NULL, &hints, &addrs);
+ Assert(ret == 0);
+
+ for (addr = addrs; addr; addr = addr->ai_next)
+ {
+#ifdef HAVE_UNIX_SOCKETS
+ /* Ignore AF_UNIX sockets, if any are returned. */
+ if (addr->ai_family == AF_UNIX)
+ continue;
+#endif
+
+ if (++tries > 1)
+ ereport(LOG,
+ (errmsg("trying another address for the statistics collector")));
+
+ if ((HeapCleanerSock = socket(addr->ai_family, SOCK_DGRAM, 0)) == PGINVALID_SOCKET)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not create socket for relation cleaner: %m")));
+ continue;
+ }
+
+ if (bind(HeapCleanerSock, addr->ai_addr, addr->ai_addrlen) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not bind socket for relation cleaner: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ alen = sizeof(HeapCleanerSockAddr);
+ if (getsockname(HeapCleanerSock, (struct sockaddr *) &HeapCleanerSockAddr, &alen) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not get address of socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ if (connect(HeapCleanerSock, (struct sockaddr *) &HeapCleanerSockAddr, alen) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not connect socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+ /*
+ * Try to send and receive a one-byte test message on the socket. This
+ * is to catch situations where the socket can be created but will not
+ * actually pass data (for instance, because kernel packet filtering
+ * rules prevent it).
+ */
+ test_byte = TESTBYTEVAL;
+
+retry1:
+ if (send(HeapCleanerSock, &test_byte, 1, 0) != 1)
+ {
+ if (errno == EINTR)
+ goto retry1; /* if interrupted, just retry */
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not send test message on socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ /*
+ * There could possibly be a little delay before the message can be
+ * received. We arbitrarily allow up to half a second before deciding
+ * it's broken.
+ */
+ for (;;) /* need a loop to handle EINTR */
+ {
+ FD_ZERO(&rset);
+ FD_SET(HeapCleanerSock, &rset);
+
+ tv.tv_sec = 0;
+ tv.tv_usec = 500000;
+ sel_res = select(HeapCleanerSock + 1, &rset, NULL, NULL, &tv);
+ if (sel_res >= 0 || errno != EINTR)
+ break;
+ }
+ if (sel_res < 0)
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("select() failed in statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+ if (sel_res == 0 || !FD_ISSET(HeapCleanerSock, &rset))
+ {
+ /*
+ * This is the case we actually think is likely, so take pains to
+ * give a specific message for it.
+ *
+ * errno will not be set meaningfully here, so don't use it.
+ */
+ ereport(LOG,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("test message did not get through on socket for statistics collector")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ test_byte++; /* just make sure variable is changed */
+
+retry2:
+ if (recv(HeapCleanerSock, &test_byte, 1, 0) != 1)
+ {
+ if (errno == EINTR)
+ goto retry2; /* if interrupted, just retry */
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not receive test message on socket for statistics collector: %m")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ if (test_byte != TESTBYTEVAL) /* strictly paranoia ... */
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("incorrect test message transmission on socket for statistics collector")));
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+ continue;
+ }
+
+ /* If we get here, we have a working socket */
+
+ break;
+ }
+
+ /* Did we find a working address? */
+ if (!addr || HeapCleanerSock == PGINVALID_SOCKET)
+ goto startup_failed;
+
+ /*
+ * Set the socket to non-blocking IO. This ensures that if the collector
+ * falls behind, statistics messages will be discarded; backends won't
+ * block waiting to send messages to the collector.
+ */
+ if (!pg_set_noblock(HeapCleanerSock))
+ {
+ ereport(LOG,
+ (errcode_for_socket_access(),
+ errmsg("could not set statistics collector socket to nonblocking mode: %m")));
+ goto startup_failed;
+ }
+ pg_freeaddrinfo_all(hints.ai_family, addrs);
+
+ return;
+
+startup_failed:
+ ereport(LOG,
+ (errmsg("disabling heap cleaner for lack of working socket")));
+
+ if (addrs)
+ pg_freeaddrinfo_all(hints.ai_family, addrs);
+
+ if (HeapCleanerSock != PGINVALID_SOCKET)
+ closesocket(HeapCleanerSock);
+ HeapCleanerSock = PGINVALID_SOCKET;
+}
+
+/*
+ * Start and initialization logic of a launcher
+ */
+NON_EXEC_STATIC void
+HeapCleanerLauncherMain(int argc, char *argv[])
+{
+ sigjmp_buf local_sigjmp_buf;
+ MemoryContext bgheap_context;
+ int counter;
+
+ am_heapcleaner_launcher = true;
+
+ /*
+ * Identify myself via ps
+ */
+ init_ps_display(pgstat_get_backend_desc(B_BG_HEAPCLNR_LAUNCHER), "", "", "");
+
+ SetProcessingMode(InitProcessing);
+
+ pqsignal(SIGHUP, SIGHUP_Handler); /* set flag to read config file */
+ pqsignal(SIGINT, StatementCancelHandler);
+ pqsignal(SIGTERM, SIGTERM_Handler);
+ pqsignal(SIGQUIT, quickdie); /* hard crash time */
+ InitializeTimeouts();
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIG_IGN);
+ pqsignal(SIGFPE, FloatExceptionHandler);
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ BaseInit();
+
+#ifndef EXEC_BACKEND
+ InitProcess();
+#endif
+
+ InitPostgres(NULL, InvalidOid, NULL, InvalidOid, NULL, false);
+
+ SetProcessingMode(NormalProcessing);
+
+ /*
+ * Create a resource owner to keep track of our resources (currently only
+ * buffer pins).
+ */
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "Background cleaner");
+
+ bgheap_context = AllocSetContextCreate(TopMemoryContext,
+ "Background cleaner",
+ ALLOCSET_DEFAULT_SIZES);
+ MemoryContextSwitchTo(bgheap_context);
+
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Since not using PG_TRY, must reset error stack by hand */
+ error_context_stack = NULL;
+
+ /* Prevent interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Forget any pending QueryCancel or timeout request */
+ disable_all_timeouts(false);
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ LWLockReleaseAll();
+ pgstat_report_wait_end();
+ AbortBufferIO();
+ UnlockBuffers();
+ if (CurrentResourceOwner)
+ /* buffer pins are released here: */
+ ResourceOwnerRelease(CurrentResourceOwner,
+ RESOURCE_RELEASE_BEFORE_LOCKS,
+ false, true);
+ /* we needn't bother with the other ResourceOwnerRelease phases */
+ AtEOXact_Buffers(false);
+ AtEOXact_SMgr();
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
+
+ /*
+ * Now return to normal top-level context and clear ErrorContext for
+ * next time.
+ */
+ MemoryContextSwitchTo(bgheap_context);
+ FlushErrorState();
+
+ /* Flush any leaked data in the top-level context */
+ MemoryContextResetAndDeleteChildren(bgheap_context);
+
+ /*
+ * Make sure pgstat also considers our stat data as gone. Note: we
+ * mustn't use autovac_refresh_stats here.
+ */
+ pgstat_clear_snapshot();
+
+ /* Now we can allow interrupts again */
+ RESUME_INTERRUPTS();
+
+ if (got_SIGTERM)
+ proc_exit(0);
+ /*
+ * Sleep at least 1 second after any error. A write error is likely
+ * to be repeated, and we don't want to be filling the error logs as
+ * fast as we can.
+ */
+ pg_usleep(1000000L);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ /*
+ * Unblock signals (they were blocked when the postmaster forked us)
+ */
+ PG_SETMASK(&UnBlockSig);
+
+ wTabCtl.FillFactor = 0.75;
+ wTabCtl.ElementsMaxNum = WAITING_MESSAGES_MAX_NUM;
+ wTabCtl.ElementSize = sizeof(CleanerMessage);
+ wTabCtl.KeySize = 2 * sizeof(Oid) + sizeof(BlockNumber);
+ wTabCtl.HashFunc = CleanerMessageHashFunc;
+ wTabCtl.CompFunc = isEqualMsgs;
+
+ Assert(heapcleaner_max_workers == 10);
+ wTab = palloc0((heapcleaner_max_workers+1)*sizeof(PSHTAB));
+ for (counter = 0; counter < heapcleaner_max_workers+1; counter++)
+ wTab[counter] = SHASH_Create(wTabCtl);
+ on_shmem_exit(FreeLauncherInfo, 0);
+elog(LOG, "-> Launcher Started!");
+ main_launcher_loop();
+
+ proc_exit(0);
+}
+
+static bool
+backend_store_dirty_block(CleanerMessage *msg)
+{
+ bool found;
+ CleanerMessage *rec;
+
+ /* Initialize collector's hash table */
+ if (dblocks == NULL)
+ {
+ SHTABCTL ctl;
+
+ ctl.ElementsMaxNum = BACKEND_DIRTY_ITEMS_MAX;
+ ctl.FillFactor = 0.75;
+ ctl.ElementSize = sizeof(CleanerMessage);
+ ctl.KeySize = 2 * sizeof(Oid) + sizeof(BlockNumber);
+ ctl.HashFunc = CleanerMessageHashFunc;
+ ctl.CompFunc = isEqualMsgs;
+ dblocks = SHASH_Create(ctl);
+ }
+
+ /* Search for new position or for duplicates */
+ rec = (CleanerMessage *) SHASH_Search(dblocks, (void *) msg, SHASH_ENTER, &found);
+
+ if (!rec)
+ return true;
+
+ if (!found)
+ rec->hits = 0;
+
+ rec->xid = msg->xid;
+ rec->hits++;
+ return false;
+}
+
+/*
+ * Send a package of dirty blocks from a Backend to the Cleaner.
+ * Form the package by passing across the table. During this pass table is
+ * cleaned. Sign of clean row is (dbNode == 0) condition.
+ */
+static void
+backend_send_dirty_blocks(void)
+{
+ int rc;
+ CleanerMessage *rec;
+ CleanerMessage data[BACKEND_DIRTY_ITEMS_MAX];
+ int nitems = 0;
+
+ if (dblocks == NULL)
+ return;
+
+ if (SHASH_Entries(dblocks) == 0)
+ return;
+
+ Assert(SHASH_Entries(dblocks) <= BACKEND_DIRTY_ITEMS_MAX);
+
+ for (SHASH_SeqReset(dblocks);
+ (rec = SHASH_SeqNext(dblocks)) != NULL; )
+ {
+ Assert(rec->hits > 0);
+ Assert(nitems < BACKEND_DIRTY_ITEMS_MAX);
+ data[nitems++] = *rec;
+ }
+ SHASH_Clean(dblocks);
+
+ /* Send data to launcher.
+ * We'll retry after EINTR, but ignore all other failures
+ */
+ do
+ {
+ rc = send(HeapCleanerSock, data, nitems*sizeof(CleanerMessage), 0);
+ } while (rc < 0 && errno == EINTR);
+}
+
+/*
+ * Send a dirty block of relation to the Cleaner.
+ * Before shipping to the Cleaner dirty blocks collects in a simple hash
+ * table. Duplicates are discarded.
+ */
+void
+HeapCleanerSend(Relation relation, BlockNumber blkno)
+{
+ CleanerMessage msg;
+ MemoryContext oldMemCxt;
+
+ if (BGHeapMemCxt == NULL)
+ BGHeapMemCxt = AllocSetContextCreate(BGHeapMemCxt, "bgheap",
+ ALLOCSET_DEFAULT_SIZES);
+
+ if (RecoveryInProgress())
+ return;
+
+ if (HeapCleanerSock == PGINVALID_SOCKET)
+ return;
+
+ /*
+ * ToDo: Cleaning of system relations make instability in regression tests
+ * output order
+ */
+ if (IsSystemRelation(relation))
+ return;
+
+ /*
+ * Check the relation type.
+ * Not all relations can be cleaned now (similarly vacuum).
+ */
+ if (relation->rd_rel->relkind != RELKIND_RELATION &&
+ relation->rd_rel->relkind != RELKIND_MATVIEW &&
+ relation->rd_rel->relkind != RELKIND_TOASTVALUE &&
+ relation->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
+ {
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" --- cannot clean non-tables or special system tables",
+ RelationGetRelationName(relation))));
+ return;
+ }
+
+ if (RELATION_IS_OTHER_TEMP(relation))
+ return;
+
+ if (relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ return;
+
+ oldMemCxt = MemoryContextSwitchTo(BGHeapMemCxt);
+
+ msg.dbNode = relation->rd_node.dbNode;
+ msg.relid = RelationGetRelid(relation);
+ msg.blkno = blkno;
+ msg.xid = /*ReadNewTransactionId();*/GetCurrentTransactionIdIfAny();
+
+ if (backend_store_dirty_block(&msg))
+ backend_send_dirty_blocks();
+
+ MemoryContextSwitchTo(oldMemCxt);
+}
+
+/*
+ * Start and initialization logic of a worker
+ */
+NON_EXEC_STATIC void
+HeapCleanerWorkerMain(int argc, char *argv[])
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ am_heapcleaner_worker = true;
+
+ /* Identify myself via ps */
+ init_ps_display(pgstat_get_backend_desc(B_BG_HEAPCLNR_WORKER), "", "", "");
+
+ SetProcessingMode(InitProcessing);
+
+ pqsignal(SIGHUP, SIGHUP_Handler);
+ pqsignal(SIGINT, StatementCancelHandler);
+ pqsignal(SIGTERM, SIGTERM_Handler);
+ pqsignal(SIGQUIT, quickdie);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
+
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIGUSR2_Handler); /* Signal: Buffer contains a message */
+ pqsignal(SIGFPE, FloatExceptionHandler);
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Early initialization */
+ BaseInit();
+
+#ifndef EXEC_BACKEND
+ InitProcess();
+#endif
+
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Prevents interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ proc_exit(0);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ PG_SETMASK(&UnBlockSig);
+
+ /* Initialization steps */
+ LWLockAcquire(HeapCleanerLock, LW_EXCLUSIVE);
+ if (HeapCleanerShmem->startingWorker != NULL)
+ {
+ MyWorkerInfo = HeapCleanerShmem->startingWorker;
+ MyWorkerInfo->pid = getpid();
+ HeapCleanerShmem->startingWorker = NULL;
+
+ /* insert into the running list */
+ dlist_push_head(&HeapCleanerShmem->runningWorkers, &MyWorkerInfo->links);
+
+ on_shmem_exit(FreeWorkerInfo, 0);
+ }
+ else
+ elog(ERROR, "No Starting worker!");
+ LWLockRelease(HeapCleanerLock);
+
+ if (OidIsValid(MyWorkerInfo->dbOid))
+ {
+ char dbname[NAMEDATALEN];
+ SHTABCTL pr_ctl;
+
+ pgstat_report_heapcleaner(MyWorkerInfo->dbOid);
+ InitPostgres(NULL, MyWorkerInfo->dbOid, NULL, InvalidOid, dbname, false);
+ SetProcessingMode(NormalProcessing);
+ set_ps_display(dbname, false);
+
+ pr_ctl.ElementsMaxNum = 1000;
+ pr_ctl.FillFactor = 0.5;
+ pr_ctl.HashFunc = DefaultHashValueFunc;
+ pr_ctl.CompFunc = DefaultCompareFunc;
+ pr_ctl.ElementSize = sizeof(DirtyRelation);
+ pr_ctl.KeySize = sizeof(int32);
+
+ PrivateRelationsTable = SHASH_Create(pr_ctl);
+
+ /* Add tracking info to pgstat */
+ pgstat_progress_start_command(PROGRESS_COMMAND_CLEANER, MyWorkerInfo->dbOid);
+
+ main_worker_loop();
+ }
+ else
+ elog(ERROR, "dbid %d not valid!", MyWorkerInfo->dbOid);
+
+ proc_exit(0);
+}
+
+/*
+ * Compare messages
+ */
+static bool
+isEqualMsgs(void *arg1, void *arg2)
+{
+ CleanerMessage *msg1 = arg1;
+ CleanerMessage *msg2 = arg2;
+
+ Assert(arg1 != NULL);
+ Assert(arg2 != NULL);
+
+ if (msg1->blkno != msg2->blkno)
+ return false;
+ if (msg1->relid != msg2->relid)
+ return false;
+ if (msg1->dbNode != msg2->dbNode)
+ return false;
+
+ return true;
+}
+
+/*
+ * IsHeapCleaner functions
+ * Return whether this is either a launcher heap cleaner process or
+ * a worker process.
+ */
+bool
+IsHeapCleanerLauncherProcess(void)
+{
+ return am_heapcleaner_launcher;
+}
+
+bool
+IsHeapCleanerWorkerProcess(void)
+{
+ return am_heapcleaner_worker;
+}
+
+/*
+ * Send signal to Postmaster for launch new worker instance.
+ * Eject one node from freeWorkers list and assign to startingWorker
+ * HeapCleanerLock must be exclusive-locked
+ */
+static void
+launch_worker(Oid dbNode)
+{
+ WorkerInfo worker;
+ dlist_node *wptr;
+
+ if (dlist_is_empty(&HeapCleanerShmem->freeWorkers))
+ elog(ERROR, "NO a free slot for background cleaner worker!");
+
+ wptr = dlist_pop_head_node(&HeapCleanerShmem->freeWorkers);
+ worker = dlist_container(WorkerInfoData, links, wptr);
+ worker->dbOid = dbNode;
+ worker->launchtime = GetCurrentTimestamp();
+ worker->nitems = 0;
+ HeapCleanerShmem->startingWorker = worker;
+
+ SendPostmasterSignal(PMSIGNAL_START_HEAPCLNR_WORKER);
+}
+
+/*
+ * Return worker that initialized for dbNode database or NULL
+ * Caller need to acquire share lock on HeapCleanerLock
+ */
+static WorkerInfo
+look_for_worker(Oid dbNode)
+{
+ dlist_node *node;
+ WorkerInfo worker = NULL;
+
+ if (!dlist_is_empty(&HeapCleanerShmem->runningWorkers))
+ for (node = dlist_head_node(&HeapCleanerShmem->runningWorkers);
+ ;
+ node = dlist_next_node(&HeapCleanerShmem->runningWorkers, node))
+ {
+ if (((WorkerInfo)node)->dbOid == dbNode)
+ {
+ worker = (WorkerInfo) node;
+ break;
+ }
+ if (!dlist_has_next(&HeapCleanerShmem->runningWorkers, node))
+ break;
+ }
+
+ return worker;
+}
+
+/*
+ * Entry point of a launcher behavior logic
+ */
+static void
+main_launcher_loop()
+{
+ bool found;
+ long timeout = -1L;
+
+ Assert(HeapCleanerSock != PGINVALID_SOCKET);
+
+ /*
+ * Do work until not catch a SIGTERM signal and all blocks
+ * not shipped to workers.
+ * timeout <= 0 tells us that the launcher does not have any work right now,
+ * and it will sleep until the socket message or the system signal.
+ */
+ while (!got_SIGTERM || (timeout > 0))
+ {
+ int rc;
+ int len;
+ CleanerMessage *msg;
+ CleanerMessage table[BACKEND_DIRTY_ITEMS_MAX];
+ WorkerInfo startingWorker;
+ WorkerInfo worker;
+ dlist_node *node;
+
+ timeout = -1L;
+
+ /* Process system signals */
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ ResetLatch(MyLatch);
+
+ /* At First, receive a message from backend */
+ len = recv(HeapCleanerSock, table, BACKEND_DIRTY_ITEMS_MAX*sizeof(CleanerMessage), 0);
+
+ LWLockAcquire(HeapCleanerLock, LW_EXCLUSIVE);
+ startingWorker = HeapCleanerShmem->startingWorker;
+
+ /*
+ * Message was received from socket.
+ */
+ if (len > 0)
+ {
+ CleanerMessage *mptr;
+
+ if (len%sizeof(CleanerMessage) != 0)
+ elog(ERROR, "INCORRECT Message size!");
+
+ /*
+ * Push all messages received from a backend to general
+ * hash table with waiting messages.
+ */
+ for (mptr = table; ((char *)mptr-(char *)table) < len; mptr++)
+ {
+ worker = look_for_worker(mptr->dbNode);
+ if (worker)
+ msg = (CleanerMessage *) SHASH_Search(wTab[worker->id], mptr, SHASH_ENTER, &found);
+ else
+ msg = (CleanerMessage *) SHASH_Search(wTab[heapcleaner_max_workers], mptr, SHASH_ENTER, &found);
+
+ if (msg == NULL)
+ /*
+ * Hash table is FULL and we can't insert a message
+ */
+ stat_missed_blocks++;
+ else if (found)
+ {
+ msg->hits++;
+ msg->xid = (msg->xid > mptr->xid) ? msg->xid : mptr->xid;
+ }
+ else
+ memcpy(msg, mptr, sizeof(CleanerMessage));
+ }
+ }
+
+ /*
+ * Main waiting list parsing.
+ */
+ if (SHASH_Entries(wTab[heapcleaner_max_workers]) > 0)
+ {
+ bool startWorker = (startingWorker != NULL);
+
+ /*
+ * Pass across general waiting list.
+ * Try to send tasks to active workers.
+ */
+ for (SHASH_SeqReset(wTab[heapcleaner_max_workers]);
+ (msg = (CleanerMessage *) SHASH_SeqNext(wTab[heapcleaner_max_workers])) != NULL; )
+ {
+ worker = look_for_worker(msg->dbNode);
+
+ if (worker)
+ {
+ CleanerMessage *temp_msg = (CleanerMessage *) SHASH_Search(wTab[worker->id], (void *) msg, SHASH_ENTER, &found);
+ if (temp_msg != NULL)
+ if (found)
+ {
+ temp_msg->hits += msg->hits;
+ temp_msg->xid = (msg->xid > temp_msg->xid) ? msg->xid : temp_msg->xid;
+ }
+ else
+ memcpy(temp_msg, msg, sizeof(CleanerMessage));
+ else
+ stat_missed_blocks++;
+
+ /*
+ * Message gone to worker. delete from main waiting list.
+ */
+ temp_msg = (CleanerMessage *) SHASH_Search(wTab[heapcleaner_max_workers], (void *) msg, SHASH_REMOVE, NULL);
+ Assert(temp_msg != NULL);
+
+ continue;
+ }
+ else if (!startWorker)
+ {
+ /* Start new worker */
+ launch_worker(msg->dbNode);
+ startWorker = true;
+ }
+ }
+ }
+
+ /*
+ * Check: if we can't process all incoming messages, we need too small
+ * timeout, check latches and go to next iteration.
+ */
+ if (SHASH_Entries(wTab[heapcleaner_max_workers]) > 0)
+ timeout = 1L;
+
+ /*
+ * See waiting lists of active workers and try to send messages.
+ */
+ if (!dlist_is_empty(&HeapCleanerShmem->runningWorkers))
+ {
+ for (node = dlist_head_node(&HeapCleanerShmem->runningWorkers);
+ ;
+ node = dlist_next_node(&HeapCleanerShmem->runningWorkers, node))
+ {
+ worker = (WorkerInfo) node;
+
+ if (SHASH_Entries(wTab[worker->id]) == 0)
+ {
+ /* Check worker idle time */
+ if (TimestampDifferenceExceeds(worker->launchtime, GetCurrentTimestamp(), WORKER_IDLE_TIME_DURATION_MAX))
+ {
+ LWLockAcquire(&worker->WorkItemLock, LW_EXCLUSIVE);
+
+ /*
+ * Check: may be worker has tasks but is too lazy
+ */
+ if (worker->nitems == 0)
+ {
+ /* Shutdown the idle worker */
+ kill(worker->pid, SIGTERM);
+ dlist_delete(&worker->links);
+ dlist_push_head(&HeapCleanerShmem->freeWorkers, &worker->links);
+
+ if (!dlist_is_empty(&HeapCleanerShmem->runningWorkers))
+ node = dlist_head_node(&HeapCleanerShmem->runningWorkers);
+ else
+ {
+ LWLockRelease(&worker->WorkItemLock);
+ break;
+ }
+ }
+ LWLockRelease(&worker->WorkItemLock);
+ }
+ else if (!dlist_has_next(&HeapCleanerShmem->runningWorkers, node))
+ break;
+ else
+ continue;
+ }
+
+ /* Put list of potentially dirty blocks to the worker shared buffer */
+ LWLockAcquire(&worker->WorkItemLock, LW_EXCLUSIVE);
+
+ SHASH_SeqReset(wTab[worker->id]);
+ while (((msg = (CleanerMessage *) SHASH_SeqNext(wTab[worker->id])) != NULL) &&
+ (worker->nitems < WORKER_TASK_ITEMS_MAX))
+ {
+ void *temp_msg;
+
+ memcpy(&worker->buffer[worker->nitems++], msg, sizeof(CleanerMessage));
+ temp_msg = SHASH_Search(wTab[worker->id], (void *) msg, SHASH_REMOVE, NULL);
+ Assert(temp_msg != NULL);
+ Assert(worker->buffer[worker->nitems-1].hits > 0);
+ }
+
+ worker->launchtime = GetCurrentTimestamp();
+ LWLockRelease(&worker->WorkItemLock);
+
+ /* Worker, you have a task! */
+ kill(worker->pid, SIGUSR2);
+
+ /*
+ * If launcher has any tasks, It check signals quickly
+ * and to work on further.
+ */
+ if (SHASH_Entries(wTab[worker->id]) > 0)
+ timeout = 1L;
+
+ if (!dlist_has_next(&HeapCleanerShmem->runningWorkers, node))
+ break;
+ }
+
+ /*
+ * Launcher have'nt any immediate tasks and
+ * need to manage idle workers only
+ */
+ if (timeout < 0)
+ /* We only need to wait idle workers */
+ timeout = 100L;
+ }
+ LWLockRelease(HeapCleanerLock);
+
+ if (!got_SIGTERM)
+ {
+ int wakeEvents = WL_LATCH_SET | WL_POSTMASTER_DEATH | WL_SOCKET_READABLE;
+
+ if (timeout > 0)
+ wakeEvents |= WL_TIMEOUT;
+ /* Wait data or signals */
+ rc = WaitLatchOrSocket(MyLatch,
+ wakeEvents,
+ HeapCleanerSock, timeout,
+ WAIT_EVENT_BGHEAP_MAIN);
+ }
+
+ /* Emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ {
+ elog(LOG, "Heap Launcher exit with 1");
+ proc_exit(1);
+ }
+ }
+
+ elog(LOG, "Heap Launcher exit with 0");
+ proc_exit(0);
+}
+
+/*
+ * Entry point of a worker behavior logic
+ */
+static void
+main_worker_loop(void)
+{
+ CleanerMessage task_item[WORKER_TASK_ITEMS_MAX];
+ DirtyRelation *dirty_relation[WORKER_RELATIONS_MAX_NUM];
+ PSHTAB FreeDirtyBlocksList;
+ long timeout = -1L;
+ int dirty_relations_num = 0;
+ SHTABCTL shctl;
+
+ shctl.FillFactor = 0.75;
+ shctl.ElementsMaxNum = WORKER_DIRTYBLOCKS_MAX_NUM;
+ shctl.ElementSize = sizeof(WorkerTask);
+ shctl.KeySize = sizeof(BlockNumber);
+ shctl.HashFunc = DefaultHashValueFunc;
+ shctl.CompFunc = DefaultCompareFunc;
+
+ FreeDirtyBlocksList = SHASH_Create(shctl);
+
+ while (!got_SIGTERM || (timeout > 0))
+ {
+ int rc;
+ int incoming_items_num = 0;
+
+ timeout = -1L;
+
+ if (got_SIGHUP)
+ {
+ got_SIGHUP = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ if (got_SIGUSR2)
+ /* It is needed only for wakeup worker */
+ got_SIGUSR2 = false;
+
+ /*
+ * Move task items from shared buffer to local and release it for
+ * new data.
+ * It is introduced in accordance with the idea that shared memory
+ * buffer is smaller than internal process buffer.
+ */
+ LWLockAcquire(&MyWorkerInfo->WorkItemLock, LW_SHARED);
+ if (MyWorkerInfo->nitems > 0)
+ {
+ incoming_items_num = MyWorkerInfo->nitems;
+ memcpy(task_item, MyWorkerInfo->buffer, incoming_items_num*sizeof(CleanerMessage));
+ MyWorkerInfo->nitems = 0;
+ }
+ LWLockRelease(&MyWorkerInfo->WorkItemLock);
+
+ /*
+ * If launcher receive some task items it need to distribute between
+ * workers and waiting lists.
+ */
+ if (incoming_items_num > 0)
+ {
+ DirtyRelation *hashent;
+ bool found;
+ int i;
+
+ /* */
+ timeout = 20L;
+
+ pgstat_progress_update_param(PROGRESS_CLEANER_MISSED_BLOCKS, stat_missed_blocks);
+
+ /* Pass across items and sort by relation */
+ for (i = 0; i < incoming_items_num; i++)
+ {
+ WorkerTask *item;
+
+ hashent = SHASH_Search(PrivateRelationsTable,
+ (void *) &(task_item[i].relid), HASH_FIND, NULL);
+
+ if (hashent == NULL)
+ {
+ elog(LOG, "Add new relation: %d", task_item[i].relid);
+ if (dirty_relations_num == WORKER_RELATIONS_MAX_NUM)
+ {
+ int j;
+ int min = SHASH_Entries(dirty_relation[0]->items);
+ int pos = 0;
+
+ /*
+ * All slots for dirty relations are busy.
+ * Search for slot with smaller number of dirty blocks
+ * and replace it.
+ */
+ elog(LOG, "All dirty relation slots are busy at worker process. Start min=%d", min);
+ for (j = 1; j < dirty_relations_num; j++)
+ {
+ int nitems = SHASH_Entries(dirty_relation[j]->items);
+ if (nitems < min)
+ {
+ min = nitems;
+ pos = j;
+ }
+ }
+
+ /* Save stats about not cleaned blocks */
+ stat_missed_blocks += SHASH_Entries(dirty_relation[pos]->items);
+
+ elog(LOG, "Remove slot %d with id=%d nitems=%lu. Set new relation id=%d",pos, dirty_relation[pos]->relid, SHASH_Entries(dirty_relation[pos]->items), task_item[i].relid);
+ hashent = SHASH_Search(PrivateRelationsTable,
+ (void *) &(dirty_relation[pos]->relid), HASH_REMOVE, &found);
+
+ Assert(found);
+ Assert(hashent != NULL);
+ SHASH_Destroy(hashent->items);
+
+ /*
+ * Fill empty position by data from last position
+ */
+ dirty_relation[pos] = dirty_relation[--dirty_relations_num];
+ }
+
+ /* Insert new relid to hash table */
+ hashent = SHASH_Search(PrivateRelationsTable, (void *) &(task_item[i].relid), HASH_ENTER, &found);
+ Assert(!found);
+ Assert(hashent != NULL);
+
+ /*
+ * At a new entry we create and init list of 'dirty' blocks
+ */
+ hashent->items = SHASH_Create(shctl);
+ dirty_relation[dirty_relations_num++] = hashent;
+ pgstat_progress_update_param(PROGRESS_CLEANER_RELATIONS, SHASH_Entries(PrivateRelationsTable));
+ }
+
+ /*
+ * Relation entry found or create.
+ * Now, add an item to a waiting list.
+ */
+ item = (WorkerTask *) SHASH_Search(hashent->items, (void *) &(task_item[i].blkno), HASH_ENTER, &found);
+ if (item == NULL)
+ stat_missed_blocks++;
+ else
+ {
+ if (!found)
+ {
+ item->hits = 0;
+ item->lastXid = InvalidTransactionId;
+ }
+
+ item->hits += task_item[i].hits;
+ Assert(item->hits > 0);
+ item->lastXid = (item->lastXid > task_item[i].xid) ? item->lastXid : task_item[i].xid;
+ }
+ }
+ incoming_items_num = 0;
+ }
+
+ PG_TRY();
+ {
+ int relcounter;
+ int64 stat_tot_wait_queue_len = 0;
+
+ stat_buf_ninmem = 0;
+
+ /* Pass along dirty relations and try to clean it */
+ for (relcounter = 0; relcounter < dirty_relations_num; relcounter++)
+ {
+ if (SHASH_Entries(FreeDirtyBlocksList) > 0)
+ SHASH_Clean(FreeDirtyBlocksList);
+
+ FreeDirtyBlocksList = cleanup_relations(dirty_relation[relcounter], FreeDirtyBlocksList, got_SIGTERM);
+
+ /*
+ * Some blocks from the list may blocked by a backend.
+ * Deferred its for next cleanup attempt.
+ */
+ if ((stat_tot_wait_queue_len += SHASH_Entries(dirty_relation[relcounter]->items)) > 0)
+ timeout = 5L;
+ }
+
+ pgstat_progress_update_param(PROGRESS_CLEANER_TIMEOUT, timeout);
+ pgstat_progress_update_param(PROGRESS_CLEANER_BUF_NINMEM, stat_buf_ninmem);
+ pgstat_progress_update_param(PROGRESS_CLEANER_TOTAL_QUEUE_LENGTH, stat_tot_wait_queue_len);
+ QueryCancelPending = false;
+ }
+ PG_CATCH();
+ {
+ HOLD_INTERRUPTS();
+ EmitErrorReport();
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ RESUME_INTERRUPTS();
+ }
+ PG_END_TRY();
+
+ if (!got_SIGTERM)
+ {
+ int wakeEvents = WL_LATCH_SET | WL_POSTMASTER_DEATH;
+ if (timeout > 0)
+ wakeEvents |= WL_TIMEOUT;
+ rc = WaitLatch(MyLatch, wakeEvents, timeout, WAIT_EVENT_BGHEAP_MAIN);
+ }
+
+ ResetLatch(MyLatch);
+
+ /* Emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+
+ pgstat_progress_end_command();
+ proc_exit(0);
+}
+
+/*
+ * SIGHUP_Handler
+ */
+static void
+SIGHUP_Handler(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGHUP = true;
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * SIGTERM_Handler
+ */
+static void
+SIGTERM_Handler(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGTERM = true;
+
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * SIGUSR2_Handler
+ * Wake up a worker to read some messages from launcher
+ */
+static void
+SIGUSR2_Handler(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ got_SIGUSR2 = true;
+
+ SetLatch(MyLatch);
+
+ errno = save_errno;
+}
+
+/*
+ * Main entry point for background heap cleaner (launcher) process, to be called from the
+ * postmaster.
+ */
+int
+StartHeapCleanerLauncher(void)
+{
+ pid_t HeapCleanerLauncherPID;
+
+#ifdef EXEC_BACKEND
+ switch ((HeapCleanerLauncherPID = bghclauncher_forkexec()))
+#else
+ switch ((HeapCleanerLauncherPID = fork_process()))
+#endif
+ {
+ case -1:
+ ereport(LOG,
+ (errmsg("could not fork background heap cleaner (launcher) process: %m")));
+ return 0;
+
+#ifndef EXEC_BACKEND
+ case 0:
+ /* in postmaster child ... */
+ InitPostmasterChild();
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ HeapCleanerLauncherMain(0, NULL);
+ break;
+#endif
+ default:
+ return (int) HeapCleanerLauncherPID;
+ }
+
+ /* shouldn't get here */
+ return 0;
+}
+
+/*
+ * Main entry point for background heap cleaner (worker) process, to be called from the
+ * postmaster.
+ */
+int
+StartHeapCleanerWorker(void)
+{
+ pid_t worker_pid;
+
+#ifdef EXEC_BACKEND
+ switch ((worker_pid = hcworker_forkexec()))
+#else
+ switch ((worker_pid = fork_process()))
+#endif
+ {
+ case -1:
+ ereport(LOG,
+ (errmsg("could not fork background heap cleaner (worker) process: %m")));
+ return 0;
+
+#ifndef EXEC_BACKEND
+ case 0:
+ /* in postmaster child ... */
+ InitPostmasterChild();
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ HeapCleanerWorkerMain(0, NULL);
+ break;
+#endif
+ default:
+ return (int) worker_pid;
+ }
+
+ /* shouldn't get here */
+ return 0;
+}
+
+/*
+ * HeapCleanerShmemInit
+ * Allocate and initialize heapcleaner-related shared memory
+ */
+void
+HeapCleanerShmemInit(void)
+{
+ bool found;
+
+ HeapCleanerShmem = (HeapCleanerShmemStruct *)
+ ShmemInitStruct("HeapCleaner Data",
+ HeapCleanerShmemSize(),
+ &found);
+
+ if (!IsUnderPostmaster)
+ {
+ WorkerInfo worker;
+ int i;
+
+ Assert(!found);
+
+ dlist_init(&HeapCleanerShmem->freeWorkers);
+ dlist_init(&HeapCleanerShmem->runningWorkers);
+
+ worker = (WorkerInfo) ((char *) HeapCleanerShmem +
+ MAXALIGN(sizeof(HeapCleanerShmemStruct)));
+
+ /* initialize the WorkerInfo free list */
+ for (i = 0; i < heapcleaner_max_workers; i++)
+ {
+ worker[i].id = i;
+ LWLockInitialize(&worker[i].WorkItemLock, TrancheId);
+ dlist_push_head(&HeapCleanerShmem->freeWorkers,
+ &worker[i].links);
+ }
+ }
+ else
+ Assert(found);
+ HeapCleanerShmem->startingWorker = NULL;
+
+ TrancheId = LWLockNewTrancheId();
+ LWLockRegisterTranche(TrancheId, "heapcleaner");
+}
+
+/*
+ * HeapCleanerShmemSiz
+ * Compute space needed for heap cleaner-related shared memory
+ */
+Size
+HeapCleanerShmemSize(void)
+{
+ Size size;
+
+ /*
+ * Need the fixed struct and the array of WorkerInfoData.
+ */
+ size = sizeof(HeapCleanerShmemStruct);
+ size = MAXALIGN(size);
+ size = add_size(size, mul_size(heapcleaner_max_workers,
+ sizeof(WorkerInfoData)));
+ return size;
+}
+
+static uint64
+CleanerMessageHashFunc(void *key, uint64 size, uint64 base)
+{
+ uint64 sum = 0;
+ CleanerMessage *msg = (CleanerMessage *) key;
+
+ Assert(key != NULL);
+ Assert(size == 2 * sizeof(Oid) + sizeof(BlockNumber));
+ Assert(base > 0);
+
+ sum += DefaultHashValueFunc(&msg->dbNode, sizeof(Oid), base);
+ sum += DefaultHashValueFunc(&msg->relid, sizeof(Oid), base);
+ sum += DefaultHashValueFunc(&msg->blkno, sizeof(BlockNumber), base);
+
+ return sum%base;
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index bbe73618c7..f992307e37 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -48,6 +48,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/fork_process.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
@@ -1394,6 +1395,20 @@ pgstat_report_autovac(Oid dboid)
pgstat_send(&msg, sizeof(msg));
}
+void
+pgstat_report_heapcleaner(Oid dboid)
+{
+ PgStat_MsgAutovacStart msg;
+
+ if (pgStatSock == PGINVALID_SOCKET)
+ return;
+
+ pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_BGHEAP_START);
+ msg.m_databaseid = dboid;
+ msg.m_start_time = GetCurrentTimestamp();
+
+ pgstat_send(&msg, sizeof(msg));
+}
/* ---------
* pgstat_report_vacuum() -
@@ -2833,6 +2848,14 @@ pgstat_bestart(void)
/* Autovacuum Worker */
beentry->st_backendType = B_AUTOVAC_WORKER;
}
+ else if (IsHeapCleanerLauncherProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_LAUNCHER;
+ }
+ else if (IsHeapCleanerWorkerProcess())
+ {
+ beentry->st_backendType = B_BG_HEAPCLNR_WORKER;
+ }
else if (am_walsender)
{
/* Wal sender */
@@ -3483,6 +3506,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_AUTOVACUUM_MAIN:
event_name = "AutoVacuumMain";
break;
+ case WAIT_EVENT_BGHEAP_MAIN:
+ event_name = "BgHeapMain";
+ break;
case WAIT_EVENT_BGWRITER_HIBERNATE:
event_name = "BgWriterHibernate";
break;
@@ -4111,6 +4137,12 @@ pgstat_get_backend_desc(BackendType backendType)
case B_AUTOVAC_WORKER:
backendDesc = "autovacuum worker";
break;
+ case B_BG_HEAPCLNR_LAUNCHER:
+ backendDesc = "heap cleaner launcher";
+ break;
+ case B_BG_HEAPCLNR_WORKER:
+ backendDesc = "heap cleaner worker";
+ break;
case B_BACKEND:
backendDesc = "client backend";
break;
@@ -4422,6 +4454,10 @@ PgstatCollectorMain(int argc, char *argv[])
pgstat_recv_autovac((PgStat_MsgAutovacStart *) &msg, len);
break;
+ case PGSTAT_MTYPE_BGHEAP_START:
+// pgstat_recv_autovac((PgStat_MsgHeapCleanerStart *) &msg, len);
+ break;
+
case PGSTAT_MTYPE_VACUUM:
pgstat_recv_vacuum((PgStat_MsgVacuum *) &msg, len);
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b33cd..d14c1b557d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -109,6 +109,7 @@
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
@@ -145,9 +146,10 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_BGHEAP 0x0010 /* autovacuum worker process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
-#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
+#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP)
/*
* List of active backends (or child processes anyway; we don't actually
@@ -252,6 +254,7 @@ static pid_t StartupPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
+ HeapClnrPID = 0,
PgArchPID = 0,
PgStatPID = 0,
SysLoggerPID = 0;
@@ -355,6 +358,7 @@ bool redirection_done = false; /* stderr redirected for syslogger? */
/* received START_AUTOVAC_LAUNCHER signal */
static volatile sig_atomic_t start_autovac_launcher = false;
+static volatile sig_atomic_t start_heapclnr_launcher = false;
/* the launcher needs to be signalled to communicate some condition */
static volatile bool avlauncher_needs_signal = false;
@@ -432,6 +436,7 @@ static void maybe_start_bgworkers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
+static void StartBgHeapWorker(void);
static void MaybeStartWalReceiver(void);
static void InitPostmasterDeathWatchHandle(void);
@@ -1310,6 +1315,7 @@ PostmasterMain(int argc, char *argv[])
*/
autovac_init();
+ HeapCleanerInit();
/*
* Load configuration files for client authentication.
*/
@@ -1757,6 +1763,14 @@ ServerLoop(void)
start_autovac_launcher = false; /* signal processed */
}
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && start_heapclnr_launcher &&
+ pmState == PM_RUN)
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ if (HeapClnrPID != 0)
+ start_heapclnr_launcher = false; /* signal processed */
+ }
+
/* If we have lost the stats collector, try to start a new one */
if (PgStatPID == 0 &&
(pmState == PM_RUN || pmState == PM_HOT_STANDBY))
@@ -1774,6 +1788,12 @@ ServerLoop(void)
kill(AutoVacPID, SIGUSR2);
}
+ /* If we have lost the background heap cleaner, try to start a new one. */
+ if (!IsBinaryUpgrade && HeapClnrPID == 0 && pmState == PM_RUN )
+ {
+ HeapClnrPID = StartHeapCleanerLauncher();
+ }
+
/* If we need to start a WAL receiver, try to do that now */
if (WalReceiverRequested)
MaybeStartWalReceiver();
@@ -2541,6 +2561,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGHUP);
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGHUP);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGHUP);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGHUP);
if (SysLoggerPID != 0)
@@ -2627,10 +2649,12 @@ pmdie(SIGNAL_ARGS)
/* autovac workers are told to shut down immediately */
/* and bgworkers too; does this need tweaking? */
SignalSomeChildren(SIGTERM,
- BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the bgwriter too */
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
@@ -2687,7 +2711,7 @@ pmdie(SIGNAL_ARGS)
signal_child(WalReceiverPID, SIGTERM);
if (pmState == PM_RECOVERY)
{
- SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER);
+ SignalSomeChildren(SIGTERM, BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/*
* Only startup, bgwriter, walreceiver, possibly bgworkers,
@@ -2708,10 +2732,12 @@ pmdie(SIGNAL_ARGS)
/* shut down all backends and workers */
SignalSomeChildren(SIGTERM,
BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC |
- BACKEND_TYPE_BGWORKER);
+ BACKEND_TYPE_BGWORKER | BACKEND_TYPE_BGHEAP);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, SIGTERM);
/* and the walwriter too */
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGTERM);
@@ -2875,6 +2901,8 @@ reaper(SIGNAL_ARGS)
*/
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
+ if (!IsBinaryUpgrade && (HeapClnrPID == 0))
+ HeapClnrPID = StartHeapCleanerLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
PgArchPID = pgarch_start();
if (PgStatPID == 0)
@@ -3009,6 +3037,14 @@ reaper(SIGNAL_ARGS)
_("autovacuum launcher process"));
continue;
}
+ if (pid == HeapClnrPID)
+ {
+ HeapClnrPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("heap cleaner launcher process"));
+ continue;
+ }
/*
* Was it the archiver? If so, just try to start a new one; no need
@@ -3475,6 +3511,17 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ if (pid == HeapClnrPID)
+ HeapClnrPID = 0;
+ else if (HeapClnrPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ "SIGQUIT",
+ (int) HeapClnrPID)));
+ signal_child(HeapClnrPID, SIGQUIT);
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3661,6 +3708,7 @@ PostmasterStateMachine(void)
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
WalWriterPID == 0 &&
+ HeapClnrPID == 0 &&
AutoVacPID == 0)
{
if (Shutdown >= ImmediateShutdown || FatalError)
@@ -3759,6 +3807,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(HeapClnrPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -3948,6 +3997,8 @@ TerminateChildren(int signal)
signal_child(WalReceiverPID, signal);
if (AutoVacPID != 0)
signal_child(AutoVacPID, signal);
+ if (HeapClnrPID != 0)
+ signal_child(HeapClnrPID, signal);
if (PgArchPID != 0)
signal_child(PgArchPID, signal);
if (PgStatPID != 0)
@@ -4781,6 +4832,7 @@ SubPostmasterMain(int argc, char *argv[])
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
+ strcmp(argv[1], "--forkbgheap") == 0 ||
strcmp(argv[1], "--forkboot") == 0 ||
strncmp(argv[1], "--forkbgworker=", 15) == 0)
PGSharedMemoryReAttach();
@@ -4792,6 +4844,8 @@ SubPostmasterMain(int argc, char *argv[])
AutovacuumLauncherIAm();
if (strcmp(argv[1], "--forkavworker") == 0)
AutovacuumWorkerIAm();
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ BgHeapCleanerIAm();
/*
* Start our win32 signal implementation. This has to be done after we
@@ -4921,6 +4975,19 @@ SubPostmasterMain(int argc, char *argv[])
AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkbgheap") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores(false, 0);
+
+ BgHeapMain(argc, argv); /* does not return */
+ }
if (strncmp(argv[1], "--forkbgworker=", 15) == 0)
{
int shmem_slot;
@@ -5122,6 +5189,12 @@ sigusr1_handler(SIGNAL_ARGS)
start_autovac_launcher = true;
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_LAUNCHER) &&
+ Shutdown == NoShutdown)
+ {
+ start_heapclnr_launcher = true;
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_AUTOVAC_WORKER) &&
Shutdown == NoShutdown)
{
@@ -5129,6 +5202,13 @@ sigusr1_handler(SIGNAL_ARGS)
StartAutovacuumWorker();
}
+ if (CheckPostmasterSignal(PMSIGNAL_START_HEAPCLNR_WORKER) &&
+ Shutdown == NoShutdown)
+ {
+ /* The autovacuum launcher wants us to start a worker process. */
+ StartBgHeapWorker();
+ }
+
if (CheckPostmasterSignal(PMSIGNAL_START_WALRECEIVER))
{
/* Startup Process wants us to start the walreceiver process. */
@@ -5470,6 +5550,73 @@ StartAutovacuumWorker(void)
}
}
+static void
+StartBgHeapWorker(void)
+{
+ Backend *bn;
+
+ /*
+ * If not in condition to run a process, don't try, but handle it like a
+ * fork failure. This does not normally happen, since the signal is only
+ * supposed to be sent by autovacuum launcher when it's OK to do it, but
+ * we have to check to avoid race-condition problems during DB state
+ * changes.
+ */
+ if (canAcceptConnections() == CAC_OK)
+ {
+ /*
+ * Compute the cancel key that will be assigned to this session. We
+ * probably don't need cancel keys for autovac workers, but we'd
+ * better have something random in the field to prevent unfriendly
+ * people from sending cancels to them.
+ */
+ if (!RandomCancelKey(&MyCancelKey))
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random cancel key")));
+ return;
+ }
+
+ bn = (Backend *) malloc(sizeof(Backend));
+ if (bn)
+ {
+ bn->cancel_key = MyCancelKey;
+
+ /* Autovac workers are not dead_end and need a child slot */
+ bn->dead_end = false;
+ bn->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+ bn->bgworker_notify = false;
+
+ bn->pid = StartHeapCleanerWorker();
+
+ if (bn->pid > 0)
+ {
+ bn->bkend_type = BACKEND_TYPE_BGHEAP;
+ dlist_push_head(&BackendList, &bn->elem);
+#ifdef EXEC_BACKEND
+ ShmemBackendArrayAdd(bn);
+#endif
+ /* all OK */
+ return;
+ }
+
+ /*
+ * fork failed, fall through to report -- actual error message was
+ * logged by StartAutoVacWorker
+ */
+ (void) ReleasePostmasterChildSlot(bn->child_slot);
+ free(bn);
+ }
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+ }
+
+ elog(ERROR, "Some BGHEAP Worker start error!");
+}
+
/*
* MaybeStartWalReceiver
* Start the WAL receiver process, if not running and our state allows.
@@ -5534,7 +5681,8 @@ CreateOptsFile(int argc, char *argv[], char *fullprogname)
int
MaxLivePostmasterChildren(void)
{
- return 2 * (MaxConnections + autovacuum_max_workers + 1 +
+ return 2 * (MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 +
max_worker_processes);
}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 01eabe5706..175902e0d3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -449,6 +449,7 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
+ bool onlyInMemory,
bool *foundPtr);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
static void AtProcExit_Buffers(int code, Datum arg);
@@ -717,6 +718,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
isExtend = (blockNum == P_NEW);
+ Assert(!(isExtend && mode == RBM_NORMAL_NO_READ));
+
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
smgr->smgr_rnode.node.spcNode,
smgr->smgr_rnode.node.dbNode,
@@ -743,9 +746,11 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, (mode == RBM_NORMAL_NO_READ), &found);
if (found)
pgBufferUsage.shared_blks_hit++;
+ else if (mode == RBM_NORMAL_NO_READ)
+ return InvalidBuffer;
else
pgBufferUsage.shared_blks_read++;
}
@@ -788,6 +793,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
return BufferDescriptorGetBuffer(bufHdr);
}
+ Assert(mode != RBM_NORMAL_NO_READ);
+
/*
* We get here only in the corner case where we are trying to extend
* the relation but we found a pre-existing buffer marked BM_VALID.
@@ -987,6 +994,7 @@ static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
+ bool onlyInMemory,
bool *foundPtr)
{
BufferTag newTag; /* identity of requested block */
@@ -1036,7 +1044,17 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* own read attempt if the page is still not BM_VALID.
* StartBufferIO does it all.
*/
- if (StartBufferIO(buf, true))
+ if (onlyInMemory == true)
+ {
+ /*
+ * We cannot get buffer immediately, go away if caller just
+ * tried to get access to already in-memory page
+ */
+ UnpinBuffer(buf, true);
+ *foundPtr = false;
+ return NULL;
+ }
+ else if (StartBufferIO(buf, true))
{
/*
* If we get here, previous attempts to read the buffer must
@@ -1055,6 +1073,13 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ /* will not read a page from disk */
+ if (onlyInMemory == true)
+ {
+ *foundPtr = false;
+ return NULL;
+ }
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1496,6 +1521,37 @@ MarkBufferDirty(Buffer buffer)
}
}
+/*
+ * IsBufferDirty - is buffer marked dirty?
+ *
+ * Buffer should be pinned, if caller wishes to get consistent result,
+ * buffer should be exclusively locked.
+ */
+bool
+IsBufferDirty(Buffer buffer)
+{
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ if (!BufferIsValid(buffer))
+ elog(ERROR, "bad buffer ID: %d", buffer);
+
+ Assert(BufferIsPinned(buffer));
+
+ if (BufferIsLocal(buffer))
+ return IsLocalBufferDirty(buffer);
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * if buffer is not exclusively locked, buf_state could be out of sync
+ * already...
+ */
+ return (buf_state & BM_DIRTY) ? true : false;
+}
+
/*
* ReleaseAndReadBuffer -- combine ReleaseBuffer() and ReadBuffer()
*
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index e4146a260a..856955635e 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -305,6 +305,30 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * IsLocalBufferDirty
+ * returns true of buffer is dirty
+ */
+bool
+IsLocalBufferDirty(Buffer buffer)
+{
+ int bufid;
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ Assert(BufferIsLocal(buffer));
+
+ bufid = -(buffer + 1);
+
+ Assert(LocalRefCount[bufid] > 0);
+
+ bufHdr = GetLocalBufferDescriptor(bufid);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ return (buf_state & BM_DIRTY) != 0;
+}
+
/*
* DropRelFileNodeLocalBuffers
* This function removes from the buffer pool all the pages of the
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 0c86a581c0..fba74752a9 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -25,6 +25,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
@@ -140,6 +141,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
size = add_size(size, ProcSignalShmemSize());
size = add_size(size, CheckpointerShmemSize());
size = add_size(size, AutoVacuumShmemSize());
+ size = add_size(size, HeapCleanerShmemSize());
size = add_size(size, ReplicationSlotsShmemSize());
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
@@ -256,6 +258,7 @@ CreateSharedMemoryAndSemaphores(bool makePrivate, int port)
ProcSignalShmemInit();
CheckpointerShmemInit();
AutoVacuumShmemInit();
+ HeapCleanerShmemInit();
ReplicationSlotsShmemInit();
ReplicationOriginShmemInit();
WalSndShmemInit();
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index e6025ecedb..73a9d141b0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -50,3 +50,4 @@ OldSnapshotTimeMapLock 42
BackendRandomLock 43
LogicalRepWorkerLock 44
CLogTruncationLock 45
+HeapCleanerLock 46
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 6f30e082b2..ecf14647b4 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
@@ -352,7 +353,7 @@ InitProcess(void)
* cleaning up. (XXX autovac launcher currently doesn't participate in
* this; it probably should.)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildActive();
/*
@@ -896,7 +897,7 @@ ProcKill(int code, Datum arg)
* way, so tell the postmaster we've cleaned up acceptably well. (XXX
* autovac launcher should be included here someday)
*/
- if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && !IsHeapCleanerLauncherProcess())
MarkPostmasterChildInactive();
/* wake autovac launcher if needed -- see comments in FreeWorkerInfo */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f4133953be..87779bd178 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -56,6 +56,7 @@
#include "parser/parser.h"
#include "pg_getopt.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -2889,6 +2890,10 @@ ProcessInterrupts(void)
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating autovacuum process due to administrator command")));
+ else if (IsHeapCleanerWorkerProcess())
+ ereport(FATAL,
+ (errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("terminating bgheap process due to administrator command")));
else if (IsLogicalWorker())
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
@@ -3017,6 +3022,13 @@ ProcessInterrupts(void)
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("canceling autovacuum task")));
}
+ if (IsHeapCleanerWorkerProcess())
+ {
+ LockErrorCleanup();
+ ereport(ERROR,
+ (errcode(ERRCODE_QUERY_CANCELED),
+ errmsg("canceling bgheap worker task")));
+ }
if (RecoveryConflictPending)
{
RecoveryConflictPending = false;
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index e95e347184..8831d98460 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
cmdtype = PROGRESS_COMMAND_VACUUM;
+ else if (pg_strcasecmp(cmd, "CLEANER") == 0)
+ cmdtype = PROGRESS_COMMAND_CLEANER;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/backend/utils/hash/Makefile b/src/backend/utils/hash/Makefile
index 64eebd1d99..9a3392fe37 100644
--- a/src/backend/utils/hash/Makefile
+++ b/src/backend/utils/hash/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/hash
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = dynahash.o hashfn.o pg_crc.o
+OBJS = dynahash.o hashfn.o pg_crc.o shash.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/hash/shash.c b/src/backend/utils/hash/shash.c
new file mode 100644
index 0000000000..9710485c85
--- /dev/null
+++ b/src/backend/utils/hash/shash.c
@@ -0,0 +1,265 @@
+/*
+ * shash.c
+ *
+ * Lightweight hash tables without dynamic expansion
+ *
+ *
+ * Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/hash/shash.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "limits.h"
+#include "utils/shash.h"
+
+#define ELEM(index) (&shtab->Elements[index*shtab->Header.ElementSize])
+
+static void check(SHTAB* shtab);
+
+/*
+ * Check data allocation of hash table variables
+ */
+static void
+check(SHTAB* shtab)
+{
+ Assert(shtab != NULL);
+ Assert(shtab->Elements != NULL);
+ Assert(shtab->state != NULL);
+}
+
+/*
+ * Create and initialize hash table with shctl options
+ */
+SHTAB*
+SHASH_Create(SHTABCTL shctl)
+{
+ SHTAB* shtab = (SHTAB *) palloc(sizeof(SHTAB));
+
+ Assert(shctl.ElementSize != 0);
+ Assert(shctl.ElementSize >= shctl.KeySize);
+
+ shtab->Header = shctl;
+ shtab->HTableSize = shtab->Header.ElementsMaxNum * (2. - shtab->Header.FillFactor) + 1;
+ Assert(shtab->HTableSize > shtab->Header.ElementsMaxNum);
+
+ /* Add one element as sign of empty value */
+ shtab->Elements = (char *) palloc(shtab->HTableSize * shctl.ElementSize);
+ shtab->state = (HESTATE *) palloc0(shtab->HTableSize * sizeof(HESTATE));
+ shtab->nElements = 0;
+ shtab->SeqScanCurElem = 0;
+
+ return shtab;
+}
+
+/*
+ * Fast cleaning of hash table.
+ * After call of this function hash table contains 0 elements
+ * and can be used again.
+ */
+void
+SHASH_Clean(SHTAB* shtab)
+{
+ int i;
+
+ check(shtab);
+
+ /*
+ * We do not free memory: only clean 'used' field and set number of elems at 0
+ */
+ for (i = 0; i < shtab->HTableSize; i++)
+ shtab->state[i] = SHASH_NUSED;
+ shtab->nElements = 0;
+ shtab->SeqScanCurElem = 0;
+}
+
+/*
+ * Free all hash table memory resources.
+ */
+void
+SHASH_Destroy(SHTAB* shtab)
+{
+ check(shtab);
+
+ pfree(shtab->Elements);
+ pfree(shtab->state);
+ pfree(shtab);
+}
+
+/*
+ * Get number of elements in the hash table.
+ */
+uint64
+SHASH_Entries(SHTAB* shtab)
+{
+ check(shtab);
+
+ Assert((shtab->nElements >= 0) && (shtab->nElements <= shtab->Header.ElementsMaxNum));
+ return shtab->nElements;
+}
+
+/*
+ * Find, enter or delete element.
+ * return NULL, if element not found.
+ */
+void*
+SHASH_Search(SHTAB* shtab, void *keyPtr, SHASHACTION action, bool *foundPtr)
+{
+ uint64 index,
+ first,
+ first_removed_index = ULONG_MAX;
+ void *result = NULL;
+
+ check(shtab);
+
+ Assert(shtab->nElements <= shtab->Header.ElementsMaxNum);
+
+ first = index = shtab->Header.HashFunc(keyPtr, shtab->Header.KeySize, shtab->HTableSize);
+ Assert(index < shtab->HTableSize);
+
+ if (foundPtr != NULL)
+ *foundPtr = false;
+
+ /*
+ * Main hash table search cycle
+ */
+ for(;;)
+ {
+ if (shtab->state[index] == SHASH_NUSED)
+ {
+ /* Empty position found */
+ switch (action)
+ {
+ case SHASH_ENTER:
+ if (shtab->nElements < shtab->Header.ElementsMaxNum)
+ {
+ if (first_removed_index != ULONG_MAX)
+ index= first_removed_index;
+
+ memcpy(ELEM(index), keyPtr, shtab->Header.KeySize);
+ shtab->state[index] = SHASH_USED;
+ shtab->nElements++;
+ result = ELEM(index);
+ }
+ break;
+ case SHASH_FIND:
+ /* Hash table not contains record with such key */
+ case SHASH_REMOVE:
+ /* may be potential problem */
+ break;
+ default:
+ Assert(0);
+ }
+
+ return result;
+ }
+
+ if ((shtab->state[index] == SHASH_USED) && (shtab->Header.CompFunc(keyPtr, ELEM(index))))
+ {
+ /*
+ * We found the element exactly
+ */
+ if (foundPtr != NULL)
+ *foundPtr = true;
+
+ if (action == SHASH_REMOVE)
+ {
+ Assert(shtab->nElements > 0);
+ /* Data will be cleaned by ENTER operation */
+ shtab->state[index] = SHASH_REMOVED;
+ shtab->nElements--;
+ }
+
+ return ELEM(index);
+ }
+
+ if ((shtab->state[index] == SHASH_REMOVED) && (action == SHASH_ENTER))
+ /* We can use this element for SHASH_ENTER, potentially */
+ if (first_removed_index == ULONG_MAX)
+ first_removed_index = index;
+
+ /* Go to next element */
+ index = (index+1)%(shtab->HTableSize);
+
+ if (index == first)
+ {
+ /*
+ * We made all-table search cycle.
+ */
+ if ((action == SHASH_ENTER) && (first_removed_index != ULONG_MAX) && (shtab->nElements < shtab->Header.ElementsMaxNum))
+ {
+ index = first_removed_index;
+ memcpy(ELEM(index), keyPtr, shtab->Header.KeySize);
+ shtab->state[index] = SHASH_USED;
+ shtab->nElements++;
+ result = ELEM(index);
+ }
+ else
+ result = NULL;
+ return result;
+ }
+ }
+
+ Assert(0);
+ return NULL;
+}
+
+/*
+ * Begin new sequental scan of the hash table.
+ */
+void
+SHASH_SeqReset(SHTAB* shtab)
+{
+ check(shtab);
+
+ shtab->SeqScanCurElem = 0;
+}
+
+/*
+ * Get next hash table value during sequental scan.
+ * Call after last element returned returns NULL.
+ */
+void*
+SHASH_SeqNext(SHTAB* shtab)
+{
+ check(shtab);
+
+ if (shtab->SeqScanCurElem == shtab->HTableSize)
+ return NULL;
+
+ Assert(shtab->SeqScanCurElem < shtab->HTableSize);
+
+ do
+ {
+ if (shtab->state[shtab->SeqScanCurElem] == SHASH_USED)
+ return ELEM(shtab->SeqScanCurElem++);
+ else
+ shtab->SeqScanCurElem++;
+ }
+ while (shtab->SeqScanCurElem < shtab->HTableSize);
+
+ return NULL;
+}
+
+/*
+ * Simple hash function.
+ */
+uint64
+DefaultHashValueFunc(void *key, uint64 size, uint64 base)
+{
+ uint64 sum = 0;
+ int i;
+
+ for (i = 0; i < size; i++)
+ {
+ uint64 x = ((char *)key)[i];
+ sum += x*x*x*x + x*x*x + x*x + x + 1;
+ }
+
+ return sum%base;
+}
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 865119d272..9ebdd1c7de 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -689,7 +690,7 @@ InitializeSessionUserIdStandalone(void)
* This function should only be called in single-user mode, in autovacuum
* workers, and in background workers.
*/
- AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker);
+ AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker || IsHeapCleanerWorkerProcess());
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 5ef6315d20..4345b8bbbc 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#include "postmaster/bgheap.h"
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
@@ -321,7 +322,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
*
* We do not enforce them for autovacuum worker processes either.
*/
- if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess())
+ if (IsUnderPostmaster && !IsAutoVacuumWorkerProcess() && !IsHeapCleanerWorkerProcess())
{
/*
* Check that the database is currently allowing connections.
@@ -504,8 +505,8 @@ InitializeMaxBackends(void)
Assert(MaxBackends == 0);
/* the extra unit accounts for the autovacuum launcher */
- MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
- max_worker_processes;
+ MaxBackends = MaxConnections + autovacuum_max_workers +
+ heapcleaner_max_workers + 1 + max_worker_processes;
/* internal error because the values were all checked previously */
if (MaxBackends > MAX_BACKENDS)
@@ -682,7 +683,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
before_shmem_exit(ShutdownPostgres, 0);
/* The autovacuum launcher is done here */
- if (IsAutoVacuumLauncherProcess())
+ if (IsAutoVacuumLauncherProcess() || IsHeapCleanerLauncherProcess())
{
/* report this backend in the PgBackendStatus array */
pgstat_bestart();
@@ -722,7 +723,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
* In standalone mode and in autovacuum worker processes, we use a fixed
* ID, otherwise we figure it out from the authenticated user name.
*/
- if (bootstrap || IsAutoVacuumWorkerProcess())
+ if (bootstrap || IsAutoVacuumWorkerProcess() || IsHeapCleanerWorkerProcess())
{
InitializeSessionUserIdStandalone();
am_superuser = true;
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 6a6b467fee..698b007d65 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,16 @@
#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+/* Progress parameters for cleaner */
+#define PROGRESS_CLEANER_RELATIONS 0
+#define PROGRESS_CLEANER_TOTAL_QUEUE_LENGTH 1
+#define PROGRESS_CLEANER_BUF_NINMEM 2
+#define PROGRESS_CLEANER_TOTAL_DELETIONS 3
+#define PROGRESS_CLEANER_VAINLY_CLEANED_TUPLES 4
+#define PROGRESS_CLEANER_MISSED_BLOCKS 5
+#define PROGRESS_CLEANER_CLEANED_BLOCKS 6
+#define PROGRESS_CLEANER_VAINLY_CLEANED_BLOCKS 7
+#define PROGRESS_CLEANER_NACQUIRED_LOCKS 8
+#define PROGRESS_CLEANER_TIMEOUT 9
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 85d472f0a5..705120eddc 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -186,6 +186,9 @@ extern void vacuum_set_xid_limits(Relation rel,
extern void vac_update_datfrozenxid(void);
extern void vacuum_delay_point(void);
+extern void quick_vacuum_index(Relation irel, Relation hrel,
+ ItemPointer dead_tuples,
+ int num_dead_tuples);
/* in commands/vacuumlazy.c */
extern void lazy_vacuum_rel(Relation onerel, int options,
VacuumParams *params, BufferAccessStrategy bstrategy);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d59c24ae23..dbf09328ac 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -56,6 +56,7 @@ typedef enum StatMsgType
PGSTAT_MTYPE_RESETSHAREDCOUNTER,
PGSTAT_MTYPE_RESETSINGLECOUNTER,
PGSTAT_MTYPE_AUTOVAC_START,
+ PGSTAT_MTYPE_BGHEAP_START,
PGSTAT_MTYPE_VACUUM,
PGSTAT_MTYPE_ANALYZE,
PGSTAT_MTYPE_ARCHIVER,
@@ -354,6 +355,12 @@ typedef struct PgStat_MsgAutovacStart
TimestampTz m_start_time;
} PgStat_MsgAutovacStart;
+typedef struct PgStat_MsgHeapCleanerStart
+{
+ PgStat_MsgHdr m_hdr;
+ Oid m_databaseid;
+ TimestampTz m_start_time;
+} PgStat_MsgHeapCleanerStart;
/* ----------
* PgStat_MsgVacuum Sent by the backend or autovacuum daemon
@@ -704,6 +711,8 @@ typedef enum BackendType
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
+ B_BG_HEAPCLNR_LAUNCHER,
+ B_BG_HEAPCLNR_WORKER,
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
@@ -756,6 +765,7 @@ typedef enum
{
WAIT_EVENT_ARCHIVER_MAIN = PG_WAIT_ACTIVITY,
WAIT_EVENT_AUTOVACUUM_MAIN,
+ WAIT_EVENT_BGHEAP_MAIN,
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
@@ -933,7 +943,8 @@ typedef enum
typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
- PROGRESS_COMMAND_VACUUM
+ PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_CLEANER
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 10
@@ -1181,6 +1192,7 @@ extern void pgstat_reset_shared_counters(const char *);
extern void pgstat_reset_single_counter(Oid objectid, PgStat_Single_Reset_Type type);
extern void pgstat_report_autovac(Oid dboid);
+extern void pgstat_report_heapcleaner(Oid dboid);
extern void pgstat_report_vacuum(Oid tableoid, bool shared,
PgStat_Counter livetuples, PgStat_Counter deadtuples);
extern void pgstat_report_analyze(Relation rel,
diff --git a/src/include/postmaster/bgheap.h b/src/include/postmaster/bgheap.h
new file mode 100644
index 0000000000..e11b310ffd
--- /dev/null
+++ b/src/include/postmaster/bgheap.h
@@ -0,0 +1,36 @@
+/*
+ * bgheap.h
+ *
+ * Created on: 27.06.2018
+ * Author: andrey
+ */
+
+#ifndef SRC_INCLUDE_ACCESS_BGHEAP_H_
+#define SRC_INCLUDE_ACCESS_BGHEAP_H_
+
+#include "storage/block.h"
+
+#ifdef EXEC_BACKEND
+extern void HeapCleanerLauncherIAm(void);
+extern void HeapCleanerWorkerIAm(void);
+extern void HeapCleanerLauncherMain(int argc, char *argv[]) pg_attribute_noreturn();
+extern void HeapCleanerWorkerMain(int argc, char *argv[]) pg_attribute_noreturn();
+#endif
+
+extern int heapcleaner_max_workers;
+
+
+extern void AtEOXact_BGHeap_tables(bool isCommit);
+extern bool IsHeapCleanerLauncherProcess(void);
+extern bool IsHeapCleanerWorkerProcess(void);
+
+extern void HeapCleanerInit(void);
+extern int StartHeapCleanerLauncher(void);
+extern int StartHeapCleanerWorker(void);
+
+extern void HeapCleanerSend(Relation relation, BlockNumber blkno);
+
+extern void HeapCleanerShmemInit(void);
+extern Size HeapCleanerShmemSize(void);
+
+#endif /* SRC_INCLUDE_ACCESS_BGHEAP_H_ */
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5370035f0c..58be48198b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -333,6 +333,7 @@ extern void LocalPrefetchBuffer(SMgrRelation smgr, ForkNumber forkNum,
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
BlockNumber blockNum, bool *foundPtr);
extern void MarkLocalBufferDirty(Buffer buffer);
+extern bool IsLocalBufferDirty(Buffer buffer);
extern void DropRelFileNodeLocalBuffers(RelFileNode rnode, ForkNumber forkNum,
BlockNumber firstDelBlock);
extern void DropRelFileNodeAllLocalBuffers(RelFileNode rnode);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 3cce3906a0..07536fc7e7 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -43,8 +43,10 @@ typedef enum
RBM_ZERO_AND_CLEANUP_LOCK, /* Like RBM_ZERO_AND_LOCK, but locks the page
* in "cleanup" mode */
RBM_ZERO_ON_ERROR, /* Read, but return an all-zeros page on error */
- RBM_NORMAL_NO_LOG /* Don't log page as invalid during WAL
+ RBM_NORMAL_NO_LOG, /* Don't log page as invalid during WAL
* replay; otherwise same as RBM_NORMAL */
+ RBM_NORMAL_NO_READ /* Don't actually read page, returns only if it
+ * presents in shared memory */
} ReadBufferMode;
/* forward declared, to avoid having to expose buf_internals.h here */
@@ -175,6 +177,7 @@ extern Buffer ReadBufferWithoutRelcache(RelFileNode rnode,
extern void ReleaseBuffer(Buffer buffer);
extern void UnlockReleaseBuffer(Buffer buffer);
extern void MarkBufferDirty(Buffer buffer);
+extern bool IsBufferDirty(Buffer buffer);
extern void IncrBufferRefCount(Buffer buffer);
extern Buffer ReleaseAndReadBuffer(Buffer buffer, Relation relation,
BlockNumber blockNum);
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 5ecc1b757c..4c73acd62c 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -36,6 +36,8 @@ typedef enum
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
+ PMSIGNAL_START_HEAPCLNR_LAUNCHER,
+ PMSIGNAL_START_HEAPCLNR_WORKER,
PMSIGNAL_BACKGROUND_WORKER_CHANGE, /* background worker state change */
PMSIGNAL_START_WALRECEIVER, /* start a walreceiver */
PMSIGNAL_ADVANCE_STATE_MACHINE, /* advance postmaster's state machine */
diff --git a/src/include/utils/shash.h b/src/include/utils/shash.h
new file mode 100644
index 0000000000..7a209cf5dc
--- /dev/null
+++ b/src/include/utils/shash.h
@@ -0,0 +1,54 @@
+#ifndef SHASH_H_
+#define SHASH_H_
+
+/* hash_search operations */
+typedef enum
+{
+ SHASH_FIND,
+ SHASH_ENTER,
+ SHASH_REMOVE
+} SHASHACTION;
+
+/* States of a hash table element */
+typedef enum
+{
+ SHASH_NUSED = 0,
+ SHASH_USED,
+ SHASH_REMOVED
+} HESTATE;
+
+typedef uint64 (*SHashValueFunc) (void *key, uint64 size, uint64 base);
+typedef bool (*CompareFunc) (void* bucket1, void* bucket2);
+
+typedef struct SHTABCTL
+{
+ uint64 ElementSize;
+ uint64 KeySize;
+ uint64 ElementsMaxNum;
+ float FillFactor;
+ SHashValueFunc HashFunc;
+ CompareFunc CompFunc;
+} SHTABCTL;
+
+typedef struct SHTAB
+{
+ SHTABCTL Header;
+ char *Elements;
+ uint64 nElements;
+ HESTATE *state; /* State of an element of hash table*/
+ uint64 SeqScanCurElem;
+ uint64 HTableSize;
+} SHTAB;
+
+typedef struct SHTAB* PSHTAB;
+
+extern PSHTAB SHASH_Create(SHTABCTL shctl);
+extern void SHASH_Clean(PSHTAB shtab);
+extern void SHASH_Destroy(PSHTAB shtab);
+extern uint64 SHASH_Entries(PSHTAB shtab);
+extern void SHASH_SeqReset(PSHTAB shtab);
+extern void* SHASH_SeqNext(PSHTAB shtab);
+extern void* SHASH_Search(PSHTAB shtab, void *keyPtr, SHASHACTION action, bool *foundPtr);
+extern uint64 DefaultHashValueFunc(void *key, uint64 size, uint64 base);
+
+#endif /* SHASH_H_ */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..478e8e698c 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -167,7 +167,7 @@ select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
a | b
---+----
1 | 11
@@ -1821,6 +1821,22 @@ pg_stat_database_conflicts| SELECT d.oid AS datid,
pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin,
pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock
FROM pg_database d;
+pg_stat_progress_cleaner| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.relid,
+ s.param1 AS relations,
+ s.param2 AS queue_len,
+ s.param3 AS ninmem,
+ s.param4 AS cleaned_tuples,
+ s.param5 AS vain_tuples,
+ s.param6 AS missed_blocks,
+ s.param7 AS cleaned_blocks,
+ s.param8 AS vain_blocks,
+ s.param9 AS nlocks,
+ s.param10 AS timeout
+ FROM (pg_stat_get_progress_info('CLEANER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index e57c6e1c42..b9c0af7bce 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -637,7 +637,7 @@ select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
i
---
3
diff --git a/src/test/regress/input/constraints.source b/src/test/regress/input/constraints.source
index 98dd4210e9..2968a1230b 100644
--- a/src/test/regress/input/constraints.source
+++ b/src/test/regress/input/constraints.source
@@ -337,7 +337,7 @@ ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- explicitly defer the constraint
BEGIN;
@@ -349,7 +349,7 @@ DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- try adding an initially deferred constraint
ALTER TABLE unique_tbl DROP CONSTRAINT unique_tbl_i_key;
@@ -367,7 +367,7 @@ DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- should fail at commit-time
BEGIN;
@@ -420,7 +420,7 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
-- test a HOT update that modifies the newly inserted tuple,
-- but should succeed because we then remove the other conflicting tuple.
@@ -431,11 +431,11 @@ INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
DROP TABLE unique_tbl;
diff --git a/src/test/regress/output/constraints.source b/src/test/regress/output/constraints.source
index a6a1df18e7..a2ae6df587 100644
--- a/src/test/regress/output/constraints.source
+++ b/src/test/regress/output/constraints.source
@@ -478,7 +478,7 @@ DETAIL: Key (i)=(1) already exists.
ROLLBACK;
-- check is done at end of statement, so this should succeed
UPDATE unique_tbl SET i = i+1;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+------
1 | one
@@ -494,14 +494,14 @@ SET CONSTRAINTS unique_tbl_i_key DEFERRED;
INSERT INTO unique_tbl VALUES (3, 'three');
DELETE FROM unique_tbl WHERE t = 'tree'; -- makes constraint valid again
COMMIT; -- should succeed
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
1 | one
2 | two
+ 3 | three
4 | four
5 | five
- 3 | three
(5 rows)
-- try adding an initially deferred constraint
@@ -516,14 +516,14 @@ UPDATE unique_tbl SET i = 2 WHERE i = 4 AND t = 'four';
DELETE FROM unique_tbl WHERE i = 1 AND t = 'one';
DELETE FROM unique_tbl WHERE i = 5 AND t = 'five';
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- should fail at commit-time
@@ -581,14 +581,14 @@ UPDATE unique_tbl SET t = 'THREE' WHERE i = 3 AND t = 'Three';
COMMIT; -- should fail
ERROR: duplicate key value violates unique constraint "unique_tbl_i_key"
DETAIL: Key (i)=(3) already exists.
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+-------
- 3 | three
1 | five
- 5 | one
- 4 | two
2 | four
+ 3 | three
+ 4 | two
+ 5 | one
(5 rows)
-- test a HOT update that modifies the newly inserted tuple,
@@ -597,25 +597,25 @@ BEGIN;
INSERT INTO unique_tbl VALUES(3, 'tree'); -- should succeed for now
UPDATE unique_tbl SET t = 'threex' WHERE t = 'tree';
DELETE FROM unique_tbl WHERE t = 'three';
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
COMMIT;
-SELECT * FROM unique_tbl;
+SELECT * FROM unique_tbl ORDER BY i;
i | t
---+--------
1 | five
- 5 | one
- 4 | two
2 | four
3 | threex
+ 4 | two
+ 5 | one
(5 rows)
DROP TABLE unique_tbl;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 3ca4c07356..c540f12a29 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -181,7 +181,7 @@ delete from rtest_v1 where b = 12;
select * from rtest_v1;
insert into rtest_v1 values (2, 12);
insert into rtest_v1 values (2, 13);
-select * from rtest_v1;
+select * from rtest_v1 ORDER BY a, b;
** Remember the delete rule on rtest_v1: It says
** DO INSTEAD DELETE FROM rtest_t1 WHERE a = old.a
** So this time both rows with a = 2 must get deleted
diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql
index 1250cd63e0..5a45fbad41 100644
--- a/src/test/regress/sql/triggers.sql
+++ b/src/test/regress/sql/triggers.sql
@@ -419,7 +419,7 @@ delete from trigtest where i=1;
select * from trigtest2;
-- ensure we still insert, even when all triggers are disabled
insert into trigtest default values;
-select * from trigtest;
+select * from trigtest ORDER BY i;
drop table trigtest2;
drop table trigtest;
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-09-20 04:35 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
3 siblings, 0 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-09-20 04:35 UTC (permalink / raw)
To: pgsql-hackers
The next version of amtargetdelete() interface and its implementation
for nbtree, devoted to the problem of retail indextuple deletion.
Uses 'v5-0001-Make-nbtree-indexes-have-unique-keys-in-tuples' patch [1]
to delete all logical duplicates by one tree descent.
[1]
https://www.postgresql.org/message-id/CAH2-WzkfK=JVHjkd17TLDvsFb6psduTt5WYiT8dg+-UFc+rSSQ@mail.gmail...
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments:
v6-0001-Retail-IndexTuple-Deletion-Access-Method.patch (text/x-patch, 13.0K) download
From 99c9bc340a730a85328549165ae8023a0716e147 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Thu, 20 Sep 2018 07:56:18 +0500
Subject: [PATCH] Retail-IndexTuple-Deletion-Access-Method
---
contrib/bloom/blutils.c | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 15 +++
src/backend/access/nbtree/nbtree.c | 160 +++++++++++++++++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/include/access/amapi.h | 6 +
src/include/access/genam.h | 18 +++
src/include/access/nbtree.h | 4 +
11 files changed, 209 insertions(+)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3742..96f1d47c70 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbcea7..a0e06bd868 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182dd7..acf14e7bec 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42effdf7..d7a53d2ca9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df30c0..5fb32d6eba 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index eade540ef5..20d9acddad 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -731,6 +731,21 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexTargetDeleteResult *
+index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation indexRelation = info->indexRelation;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, values, isnull);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index e8725fbbe1..a5d8ad4aa2 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -127,6 +127,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -886,6 +887,165 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+/*
+ * Deletion of index entries corresponding to heap TIDs.
+ *
+ * Constraints:
+ * 1. TID list info->dead_tuples arranged in ASC order.
+ * 2. Logical duplicates of index tuples stored in DESC order.
+ *
+ * The function generates an insertion scan key and descent by btree for the first
+ * index tuple what satisfies scan key and last TID in info->dead_tuples list.
+ * For the scan results it deletes all index entries, matched to the TID list.
+ *
+ * Result: a palloc'd struct containing statistical info.
+ */
+IndexTargetDeleteResult*
+bttargetdelete(IndexTargetDeleteInfo *info, IndexTargetDeleteResult *stats,
+ Datum *values, bool *isnull)
+{
+ Relation irel = info->indexRelation;
+ Relation hrel = info->heapRelation;
+ ScanKey skey;
+ int keysCount = IndexRelationGetNumberOfKeyAttributes(irel);
+ BTStack stack;
+ Buffer buf;
+ Page page;
+ BTPageOpaque opaque;
+ OffsetNumber offnum;
+ int ndeletable = 0;
+ OffsetNumber deletable[MaxOffsetNumber];
+ int pos = info->last_dead_tuple;
+ IndexTuple itup;
+
+ if (stats == NULL)
+ stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+
+ /* Assemble scankey */
+ itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
+ skey = _bt_mkscankey(irel, itup);
+
+ /* Descend the tree and position ourselves on the target leaf page. */
+ stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_WRITE, NULL);
+
+ /* trade in our read lock for a write lock */
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBuffer(buf, BT_WRITE);
+
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_WRITE, NULL);
+
+ /* To prepare tuple entries search across index pages */
+ Assert(BufferIsValid(buf));
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], P_FIRSTDATAKEY(opaque), false);
+
+ for (;;)
+ {
+ int32 cmpval;
+ ItemId itemid;
+ IndexTuple itup;
+
+ /* Switch to the next page */
+ if (offnum > PageGetMaxOffsetNumber(page))
+ {
+ /*
+ * Before unlocking index page we need to delete
+ * all currently found tuples
+ */
+ if (ndeletable > 0)
+ {
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+ stats->tuples_removed += ndeletable;
+ ndeletable = 0;
+ }
+
+ /*
+ * Check for end-of-index
+ */
+ if (P_RIGHTMOST(opaque))
+ /* it is rightmost leaf */
+ break;
+
+ /*
+ * Traverse to a next reliable index page if current page do not
+ * includes the (scan key; TID) value at the range.
+ */
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_WRITE, NULL);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ Assert(!P_IGNORE(opaque));
+
+ /* Set offnum to first potentially interesting item */
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], P_FIRSTDATAKEY(opaque), false);
+
+ if (offnum > PageGetMaxOffsetNumber(page))
+ /* Index relation do not contain TID of this DEAD tuple. */
+ break;
+ }
+
+ /* This index entry satisfied to the scan key? */
+ cmpval = _bt_compare(irel, keysCount, skey, NULL, page, offnum);
+
+ if (cmpval != 0)
+ /* End of the logical duplicates chain */
+ break;
+
+ /*
+ * To load index tuple and look for matches in the TID list of
+ * dead heap tuples
+ */
+ itemid = PageGetItemId(page, offnum);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+
+ /*
+ * Search for next TID from presorted btree result comparable
+ * to TID from presorted dead_tuples tid list
+ */
+ while (pos >= 0)
+ {
+ int res = ItemPointerCompare(&(itup->t_tid), &info->dead_tuples[pos]);
+ if ((res == 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+ else if (res > 0)
+ break;
+
+ pos--;
+ }
+
+ if (pos < 0)
+ /* End of DEAD TIDs list was reached */
+ break;
+
+ offnum = OffsetNumberNext(offnum);
+ }
+
+ /*
+ * Delete rest of dead index entries
+ */
+ if (ndeletable > 0)
+ {
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+ stats->tuples_removed += ndeletable;
+ }
+
+ /* Release stack, scan key, unpin and unlock buffer */
+ if (stack)
+ _bt_freestack(stack);
+ _bt_freeskey(skey);
+ _bt_relbuf(irel, buf);
+
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 9919e6f0d7..bdf1fdc472 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -65,6 +65,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6bb2..497b54e8a8 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,11 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexTargetDeleteResult *(*amtargetdelete_function) (IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 534fac7bf2..ee005891d2 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -51,6 +51,15 @@ typedef struct IndexVacuumInfo
BufferAccessStrategy strategy; /* access strategy for reads */
} IndexVacuumInfo;
+typedef struct IndexTargetDeleteInfo
+{
+ Relation heapRelation;
+ Relation indexRelation; /* the index being vacuumed */
+ int last_dead_tuple;
+ ItemPointer dead_tuples;
+ bool* found_dead_tuples;
+} IndexTargetDeleteInfo;
+
/*
* Struct for statistics returned by ambulkdelete and amvacuumcleanup
*
@@ -79,6 +88,11 @@ typedef struct IndexBulkDeleteResult
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+typedef struct IndexTargetDeleteResult
+{
+ int tuples_removed; /* # removed during vacuum operation */
+} IndexTargetDeleteResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
@@ -163,6 +177,10 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexTargetDeleteResult *index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 12f57773e7..59f8456e0b 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -557,6 +557,10 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexTargetDeleteResult *bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-09-21 03:52 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-11-29 13:27 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Dmitry Dolgov <9erthalion6@gmail.com>
3 siblings, 1 reply; 47+ messages in thread
From: Andrey Lepikhov @ 2018-09-21 03:52 UTC (permalink / raw)
To: pgsql-hackers
The v6 version of quick vacuum, which utilizes the amtargetdelete()
interface for retail indextuple deletion.
Now it is more simple and laconic.
It must be applied after Retail-IndexTuple-Deletion-Access-Method.patch.
BENCHMARKS:
-----------
Initial script:
pgbench -i -s $scale # initial DB generation
"CREATE INDEX pgbench_accounts_ext ON public.pgbench_accounts USING
btree (abalance);" # additional index
Comparison with lazy vacuum:
script:
"DELETE FROM pgbench_accounts WHERE (random() < $factor);" # delete a
part of tuples for cleaning strategies comparison
"VACUUM pgbench_accounts;" # check time of vacuum process by bash 'date
+%s%N | cut -b1-13' command
Results:
| $scale=10 | $scale=100 |
$factor| QVAC | LVAC | QVAC | LVAC |
1E-6 | - | - | 284 | 979 |
1E-5 | 78 | 144 | 288 | 1423 |
1E-4 | 72 | 280 | 388 | 3304 |
1E-3 | 189 | 609 | 2294 | 6029 |
1E-2 | 443 | 783 | 54232| 67884|
1E-1 | 1593 | 1237 | 83092| 86104|
where QVAC - forced use of quick vacuum; LVAC - use lazy vacuum for
index cleanup. $factor corresponds a number of vacuumed tuples. For
example, $scale=10, $factor=1E-1 -> 100000 tuples vacuumed. Time
measured in ms.
So, quick strategy can be used in a vacuum process effectively up to
1-2% of DEAD tuples in a relation.
--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments:
v6-0001-Quick-Vacuum-Strategy.patch (text/x-patch, 11.6K) download
From a5664f6fa42f615d1fb46250ac61aa9ac8a9d32d Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Thu, 20 Sep 2018 13:04:26 +0500
Subject: [PATCH] Quick Vacuum Strategy
---
src/backend/access/heap/heapam.c | 31 +++++++
src/backend/access/heap/pruneheap.c | 41 ++++++++--
src/backend/catalog/index.c | 122 ++++++++++++++++++++++++++++
src/backend/commands/vacuumlazy.c | 32 ++++++--
src/include/access/heapam.h | 2 +
src/include/catalog/index.h | 4 +
6 files changed, 220 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 56f1d82f96..0f0949bef3 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -9412,6 +9412,36 @@ heap_sync(Relation rel)
}
}
+/*
+ * Mask DEAD tuples at a HEAP page
+ *
+ * We introduce this function for wal_consistency_checking satisfaction at
+ * Hot Standby node.
+ *
+ * DEAD tuple on a master node has t_cid and t_infomask, which can not be
+ * obtained by WAL records applying on a Hot Standby node.
+ */
+static void
+mask_dead_tuples(Page page)
+{
+ OffsetNumber offnum;
+
+ for (offnum = FirstOffsetNumber; offnum <= PageGetMaxOffsetNumber(page); offnum = OffsetNumberNext(offnum))
+ {
+ ItemId lp = PageGetItemId(page, offnum);
+ char* htup_begin;
+
+ if (!ItemIdIsDead(lp))
+ continue;
+
+ if (!ItemIdHasStorage(lp))
+ continue;
+
+ htup_begin = (char *) PageGetItem(page, lp);
+ memset(htup_begin, MASK_MARKER, ItemIdGetLength(lp));
+ }
+}
+
/*
* Mask a heap page before performing consistency checks on it.
*/
@@ -9425,6 +9455,7 @@ heap_mask(char *pagedata, BlockNumber blkno)
mask_page_hint_bits(page);
mask_unused_space(page);
+ mask_dead_tuples(page);
for (off = 1; off <= PageGetMaxOffsetNumber(page); off++)
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343dac..f66e5d0260 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -405,10 +408,9 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (HeapTupleSatisfiesVacuum(&tup, OldestXmin, buffer)
== HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
{
- heap_prune_record_unused(prstate, rootoffnum);
+ heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
- ndeleted++;
}
/* Nothing more to do */
@@ -580,8 +582,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
+ if (chainitems[i] == latestdead)
+ continue;
ndeleted++;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +602,28 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ /*
+ * We allow overlapping of redirected and dead items
+ */
+ heap_prune_record_redirect(prstate, rootoffnum, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
+ }
else
+ {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -653,7 +676,12 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
+
+ /*
+ * We suppress checking prstate->marked[offnum]. It is not the best idea,
+ * but this is most simplistic way to enable Dead Redirecting by
+ * overlapping Dead and Redirected states.
+ */
prstate->marked[offnum] = true;
}
@@ -706,7 +734,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 7eb3e35166..15648c49c5 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -4160,3 +4160,125 @@ RestoreReindexState(void *reindexstate)
sistate->pendingReindexedIndexes[c]);
MemoryContextSwitchTo(oldcontext);
}
+
+
+/*
+ * htid2itup - get datum for index tuple assembly.
+ *
+ * Returns false, if Index Datum can't be generated by current TID
+ */
+bool
+htid2IndexDatum(Relation hrel, Relation irel, ItemPointer htid, Datum *values, bool *isnull)
+{
+ IndexInfo *indexInfo = BuildIndexInfo(irel);
+ EState *estate = CreateExecutorState();
+ ExprContext *econtext = GetPerTupleExprContext(estate);
+ ExprState *predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ HeapTuple tuple;
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum = ItemPointerGetOffsetNumber(htid);
+ ItemId lp;
+ BlockNumber npages;
+
+ npages = RelationGetNumberOfBlocks(hrel);
+
+ if (ItemPointerGetBlockNumber(htid) > npages)
+ return false;
+
+ buffer = ReadBuffer(hrel, ItemPointerGetBlockNumber(htid));
+ page = (Page) BufferGetPage(buffer);
+ lp = PageGetItemId(page, offnum);
+
+ Assert(ItemIdIsUsed(lp));
+
+ /* Walk along the chain */
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+ Assert(ItemIdIsUsed(lp));
+ }
+
+ /* Form a tuple */
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(hrel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ ReleaseBuffer(buffer);
+
+ econtext->ecxt_scantuple = slot;
+ MemoryContextReset(econtext->ecxt_per_tuple_memory);
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * In a partial index, ignore tuples that don't satisfy the
+ * predicate.
+ */
+ if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
+ {
+ pfree(tuple);
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+ return false;
+ }
+
+ FormIndexDatum(indexInfo, slot, estate, values, isnull);
+
+ pfree(tuple);
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+ return true;
+}
+
+/*
+ * quick_vacuum_index() -- quick vacuum one index relation.
+ *
+ * Delete all the index entries pointing to tuples listed in
+ * dead_tuples.
+ */
+void
+quick_vacuum_index(Relation irel, Relation hrel,
+ ItemPointer dead_tuples,
+ int num_dead_tuples)
+{
+ int tnum;
+ bool *found = palloc0(num_dead_tuples*sizeof(bool));
+ IndexTargetDeleteResult stats;
+ IndexTargetDeleteInfo ivinfo;
+
+ Assert(found != NULL);
+
+ stats.tuples_removed = 0;
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+
+ /* Get tuple from heap */
+ for (tnum = num_dead_tuples-1; tnum >= 0; tnum--)
+ {
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+
+ /* Index entry for the TID was deleted early */
+ if (found[tnum])
+ continue;
+
+ /* Get an index tuple building data by heap tid */
+ if (!htid2IndexDatum(hrel, irel, &(dead_tuples[tnum]), values, isnull))
+ continue;
+
+ /*
+ * Make attempt to delete some index entries by one tree descent.
+ * We use only a part of TID list, which contains not found TID's.
+ */
+ ivinfo.dead_tuples = dead_tuples;
+ ivinfo.last_dead_tuple = tnum;
+ ivinfo.found_dead_tuples = found;
+
+ index_target_delete(&ivinfo, &stats, values, isnull);
+ }
+
+ pfree(found);
+}
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 8996d366e9..f0b0a2ffb9 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -41,9 +41,11 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/nbtree.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
+#include "catalog/index.h"
#include "catalog/storage.h"
#include "commands/dbcommands.h"
#include "commands/progress.h"
@@ -744,9 +746,18 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
- &indstats[i],
- vacrelstats);
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel,
+ vacrelstats->dead_tuples,
+ vacrelstats->num_dead_tuples);
+ else
+ lazy_vacuum_index(Irel[i],
+ &indstats[i],
+ vacrelstats);
+ }
/*
* Report that we are now vacuuming the heap. We also increase
@@ -1389,10 +1400,18 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
- &indstats[i],
- vacrelstats);
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel,
+ vacrelstats->dead_tuples,
+ vacrelstats->num_dead_tuples);
+ else
+ lazy_vacuum_index(Irel[i],
+ &indstats[i],
+ vacrelstats);
+ }
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
@@ -1681,7 +1700,6 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
-
/*
* lazy_vacuum_index() -- vacuum one index relation.
*
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7497..cba8130422 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,8 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
+
/*
* HeapScanIsValid
* True iff the heap scan is valid.
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index f20c5f789b..6b1d726338 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -153,5 +153,9 @@ extern void SerializeReindexState(Size maxsize, char *start_address);
extern void RestoreReindexState(void *reindexstate);
extern void IndexSetParentIndex(Relation idx, Oid parentOid);
+extern bool htid2IndexDatum(Relation hrel, Relation irel, ItemPointer htid, Datum *values, bool *isnull);
+extern void quick_vacuum_index(Relation irel, Relation hrel,
+ ItemPointer dead_tuples,
+ int num_dead_tuples);
#endif /* INDEX_H */
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-09-21 03:52 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-11-29 13:27 ` Dmitry Dolgov <9erthalion6@gmail.com>
2019-01-31 11:20 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 47+ messages in thread
From: Dmitry Dolgov @ 2018-11-29 13:27 UTC (permalink / raw)
To: a.lepikhov@postgrespro.ru; +Cc: pgsql-hackers
> On Fri, Sep 21, 2018 at 5:52 AM Andrey Lepikhov <a.lepikhov@postgrespro.ru> wrote:
>
> The v6 version of quick vacuum, which utilizes the amtargetdelete()
> interface for retail indextuple deletion.
> Now it is more simple and laconic.
> It must be applied after Retail-IndexTuple-Deletion-Access-Method.patch.
>
> BENCHMARKS:
> -----------
>
> Initial script:
> pgbench -i -s $scale # initial DB generation
> "CREATE INDEX pgbench_accounts_ext ON public.pgbench_accounts USING
> btree (abalance);" # additional index
>
> Comparison with lazy vacuum:
>
> script:
> "DELETE FROM pgbench_accounts WHERE (random() < $factor);" # delete a
> part of tuples for cleaning strategies comparison
> "VACUUM pgbench_accounts;" # check time of vacuum process by bash 'date
> +%s%N | cut -b1-13' command
>
> Results:
> | $scale=10 | $scale=100 |
> $factor| QVAC | LVAC | QVAC | LVAC |
> 1E-6 | - | - | 284 | 979 |
> 1E-5 | 78 | 144 | 288 | 1423 |
> 1E-4 | 72 | 280 | 388 | 3304 |
> 1E-3 | 189 | 609 | 2294 | 6029 |
> 1E-2 | 443 | 783 | 54232| 67884|
> 1E-1 | 1593 | 1237 | 83092| 86104|
>
> where QVAC - forced use of quick vacuum; LVAC - use lazy vacuum for
> index cleanup. $factor corresponds a number of vacuumed tuples. For
> example, $scale=10, $factor=1E-1 -> 100000 tuples vacuumed. Time
> measured in ms.
>
> So, quick strategy can be used in a vacuum process effectively up to
> 1-2% of DEAD tuples in a relation.
Hi,
Unfortunately, this patch doesn't compile anymore:
index.c: In function ‘htid2IndexDatum’:
index.c:4172:2: error: too few arguments to function ‘MakeSingleTupleTableSlot’
TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
^
Also I'm a bit confused about the current status of collaboration between this
patch and [1], one is tightly depends on another or not? Does it makes sense
to have only one corresponding CF item then? For now I'll move this one to
the next CF.
[1]: https://www.postgresql.org/message-id/flat/CAH2-WzkVb0Kom%3DR%2B88fDFb%3DJSxZMFvbHVC6Mn9LJ2n%3DX%3Dk...
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-08-07 07:19 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-09-21 03:52 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-11-29 13:27 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Dmitry Dolgov <9erthalion6@gmail.com>
@ 2019-01-31 11:20 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 47+ messages in thread
From: Andres Freund @ 2019-01-31 11:20 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: pgsql-hackers
On 2018-11-29 14:27:50 +0100, Dmitry Dolgov wrote:
> > On Fri, Sep 21, 2018 at 5:52 AM Andrey Lepikhov <a.lepikhov@postgrespro.ru> wrote:
> >
> > The v6 version of quick vacuum, which utilizes the amtargetdelete()
> > interface for retail indextuple deletion.
> > Now it is more simple and laconic.
> > It must be applied after Retail-IndexTuple-Deletion-Access-Method.patch.
> >
> > BENCHMARKS:
> > -----------
> >
> > Initial script:
> > pgbench -i -s $scale # initial DB generation
> > "CREATE INDEX pgbench_accounts_ext ON public.pgbench_accounts USING
> > btree (abalance);" # additional index
> >
> > Comparison with lazy vacuum:
> >
> > script:
> > "DELETE FROM pgbench_accounts WHERE (random() < $factor);" # delete a
> > part of tuples for cleaning strategies comparison
> > "VACUUM pgbench_accounts;" # check time of vacuum process by bash 'date
> > +%s%N | cut -b1-13' command
> >
> > Results:
> > | $scale=10 | $scale=100 |
> > $factor| QVAC | LVAC | QVAC | LVAC |
> > 1E-6 | - | - | 284 | 979 |
> > 1E-5 | 78 | 144 | 288 | 1423 |
> > 1E-4 | 72 | 280 | 388 | 3304 |
> > 1E-3 | 189 | 609 | 2294 | 6029 |
> > 1E-2 | 443 | 783 | 54232| 67884|
> > 1E-1 | 1593 | 1237 | 83092| 86104|
> >
> > where QVAC - forced use of quick vacuum; LVAC - use lazy vacuum for
> > index cleanup. $factor corresponds a number of vacuumed tuples. For
> > example, $scale=10, $factor=1E-1 -> 100000 tuples vacuumed. Time
> > measured in ms.
> >
> > So, quick strategy can be used in a vacuum process effectively up to
> > 1-2% of DEAD tuples in a relation.
>
> Hi,
>
> Unfortunately, this patch doesn't compile anymore:
>
> index.c: In function ‘htid2IndexDatum’:
> index.c:4172:2: error: too few arguments to function ‘MakeSingleTupleTableSlot’
> TupleTableSlot *slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
> ^
>
> Also I'm a bit confused about the current status of collaboration between this
> patch and [1], one is tightly depends on another or not? Does it makes sense
> to have only one corresponding CF item then? For now I'll move this one to
> the next CF.
>
> [1]: https://www.postgresql.org/message-id/flat/CAH2-WzkVb0Kom%3DR%2B88fDFb%3DJSxZMFvbHVC6Mn9LJ2n%3DX%3Dk...
Given this patch has not been updated since Dmitry's message above, I'm
marking this as returned with feedback.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-07-23 06:25 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2 siblings, 0 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-07-23 06:25 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
This v5 version of the patch is intended to use with
v3-0001-Make-all-nbtree-index-tuples-have-unique-keys patch [1].
[1]
https://www.postgresql.org/message-id/CAH2-WzkmTRXh%3DzyMAUHyG3%3DO-QQip6CJc2VyNijRO-vzgPxmoQ%40mail...
13.07.2018 00:00, Peter Geoghegan пишет:
> On Tue, Jul 3, 2018 at 5:17 AM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> Done.
>> Attachment contains an update for use v.2 of the 'Ensure nbtree leaf tuple
>> keys are always unique' patch.
> My v3 is still pending, but is now a lot better than v2. There were
> bugs in v2 that were fixed.
>
> One area that might be worth investigating is retail index tuple
> deletion performed within the executor in the event of non-HOT
> updates. Maybe LP_REDIRECT could be repurposed to mean "ghost record",
> at least in unique index tuples with no NULL values. The idea is that
> MVCC index scans can skip over those if they've already found a
> visible tuple with the same value. Also, when there was about to be a
> page split, they could be treated a little bit like LP_DEAD items. Of
> course, the ghost bit would have to be treated as a hint that could be
> "wrong" (e.g. because the transaction hasn't committed yet), so you'd
> have to go to the heap in the context of a page split, to double
> check. Also, you'd need heuristics that let you give up on this
> strategy when it didn't help.
>
> I think that this could work well enough for OLTP workloads, and might
> be more future-proof than doing it in VACUUM. Though, of course, it's
> still very complicated.
>
--
---
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
Attachments: download all (3 files, .tar.gz)
0001-v5-Retail-IndexTuple-Deletion-Access-Method.patch (text/x-patch, 12.0K) download
From 022338178180b9968e365ed4157d35be7fb4c664 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 23 Jul 2018 09:35:17 +0500
Subject: [PATCH 1/4] v5 Retail IndexTuple Deletion Access Method
---
contrib/bloom/blutils.c | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 15 +++
src/backend/access/nbtree/nbtree.c | 138 +++++++++++++++++++++++++++
src/backend/access/spgist/spgutils.c | 1 +
src/include/access/amapi.h | 6 ++
src/include/access/genam.h | 18 ++++
src/include/access/nbtree.h | 4 +
11 files changed, 187 insertions(+)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 6b2b9e3742..96f1d47c70 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -126,6 +126,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = blbuild;
amroutine->ambuildempty = blbuildempty;
amroutine->aminsert = blinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index e95fbbcea7..a0e06bd868 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -103,6 +103,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = brinbuild;
amroutine->ambuildempty = brinbuildempty;
amroutine->aminsert = brininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 0a32182dd7..acf14e7bec 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -58,6 +58,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = ginbuild;
amroutine->ambuildempty = ginbuildempty;
amroutine->aminsert = gininsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 8a42effdf7..d7a53d2ca9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -80,6 +80,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = gistbuild;
amroutine->ambuildempty = gistbuildempty;
amroutine->aminsert = gistinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 0002df30c0..5fb32d6eba 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -76,6 +76,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 22b5cc921f..9ebeb789fb 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -730,6 +730,21 @@ index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap)
return ntids;
}
+IndexTargetDeleteResult *
+index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation indexRelation = info->indexRelation;
+
+ RELATION_CHECKS;
+
+ CHECK_REL_PROCEDURE(amtargetdelete);
+
+ return indexRelation->rd_amroutine->amtargetdelete(info, stats, values, isnull);
+}
+
/* ----------------
* index_bulk_delete - do mass deletion of index entries
*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index e8725fbbe1..44f4d0dc9f 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -34,7 +34,9 @@
#include "storage/smgr.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/syscache.h"
/* Working state needed by btvacuumpage */
@@ -127,6 +129,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amtargetdelete = bttargetdelete;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -886,6 +889,141 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
+static int
+tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
+{
+ for (int i = 0; i < ntid; i++)
+ if (ItemPointerEquals(tid, &(tid_list[i])))
+ return i;
+ return -1;
+}
+
+IndexTargetDeleteResult*
+bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull)
+{
+ Relation irel = info->indexRelation;
+ Relation hrel = info->heapRelation;
+ ScanKey skey;
+ int keysCount = IndexRelationGetNumberOfKeyAttributes(irel);
+ BTStack stack;
+ Buffer buf;
+ Page page;
+ BTPageOpaque opaque;
+ OffsetNumber offnum;
+ int ndeletable = 0;
+ OffsetNumber deletable[MaxOffsetNumber];
+ IndexTuple itup;
+
+ if (stats == NULL)
+ stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+
+ itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
+ skey = _bt_mkscankey(irel, itup);
+
+ /* Descend the tree and position ourselves on the target leaf page. */
+ stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
+ _bt_freestack(stack);
+
+ /* To prepare tuple entries search across index pages */
+ Assert(BufferIsValid(buf));
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+
+ for (;;)
+ {
+ int32 cmpval;
+ ItemId itemid;
+ IndexTuple itup;
+ int pos;
+
+ /* Switch to the next page */
+ if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ {
+ /*
+ * Before unlocking index page we need to delete
+ * all currently found tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ ndeletable = 0;
+ }
+
+ /*
+ * Check for end-of-index
+ */
+ if (P_RIGHTMOST(opaque))
+ /* it is rightmost leaf */
+ break;
+
+ /*
+ * Switch to the next index page
+ */
+ buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ page = BufferGetPage(buf);
+ _bt_checkpage(irel, buf);
+ opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = P_FIRSTDATAKEY(opaque);
+ continue;
+ }
+
+ /*
+ * This index entry satisfied to the scan key?
+ */
+ cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+
+ if (cmpval != 0)
+ /* End of index entries, satisfied to the scan key */
+ break;
+
+ /*
+ * To load index tuple and look for matches in the TID list of
+ * dead heap tuples
+ */
+ itemid = PageGetItemId(page, offnum);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
+
+ if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+
+ offnum = OffsetNumberNext(offnum);
+ }
+
+ /*
+ * Delete all found index tuples
+ */
+ if (ndeletable > 0)
+ {
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ LockBufferForCleanup(buf);
+
+ _bt_delitems_delete(irel, buf, deletable, ndeletable, hrel);
+
+ stats->tuples_removed += ndeletable;
+ }
+
+ /* Release scan key, unpin and unlock buffer */
+ _bt_freeskey(skey);
+ _bt_relbuf(irel, buf);
+
+ return stats;
+}
+
/*
* Post-VACUUM cleanup.
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 4a9b5da268..325728122b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -56,6 +56,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambuild = spgbuild;
amroutine->ambuildempty = spgbuildempty;
amroutine->aminsert = spginsert;
+ amroutine->amtargetdelete = NULL;
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 14526a6bb2..497b54e8a8 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -76,6 +76,11 @@ typedef bool (*aminsert_function) (Relation indexRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* target delete */
+typedef IndexTargetDeleteResult *(*amtargetdelete_function) (IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
@@ -207,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amtargetdelete_function amtargetdelete;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 24c720bf42..79b386eebd 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -51,6 +51,15 @@ typedef struct IndexVacuumInfo
BufferAccessStrategy strategy; /* access strategy for reads */
} IndexVacuumInfo;
+typedef struct IndexTargetDeleteInfo
+{
+ Relation heapRelation;
+ Relation indexRelation; /* the index being vacuumed */
+ int num_dead_tuples;
+ ItemPointer dead_tuples;
+ bool* found_dead_tuples;
+} IndexTargetDeleteInfo;
+
/*
* Struct for statistics returned by ambulkdelete and amvacuumcleanup
*
@@ -79,6 +88,11 @@ typedef struct IndexBulkDeleteResult
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+typedef struct IndexTargetDeleteResult
+{
+ int tuples_removed; /* # removed during vacuum operation */
+} IndexTargetDeleteResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
@@ -163,6 +177,10 @@ extern HeapTuple index_fetch_heap(IndexScanDesc scan);
extern HeapTuple index_getnext(IndexScanDesc scan, ScanDirection direction);
extern int64 index_getbitmap(IndexScanDesc scan, TIDBitmap *bitmap);
+extern IndexTargetDeleteResult *index_target_delete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 04ecb4cbc0..3b9461c3b4 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -508,6 +508,10 @@ extern void btparallelrescan(IndexScanDesc scan);
extern void btendscan(IndexScanDesc scan);
extern void btmarkpos(IndexScanDesc scan);
extern void btrestrpos(IndexScanDesc scan);
+extern IndexTargetDeleteResult *bttargetdelete(IndexTargetDeleteInfo *info,
+ IndexTargetDeleteResult *stats,
+ Datum *values,
+ bool *isnull);
extern IndexBulkDeleteResult *btbulkdelete(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
IndexBulkDeleteCallback callback,
--
2.17.1
0002-v5-Quick-Vacuum-Strategy.patch (text/x-patch, 12.3K) download
From 7b56036ea5f6817c1f81e3d53a197c50911ec69a Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 23 Jul 2018 09:36:23 +0500
Subject: [PATCH 2/4] v5 Quick Vacuum Strategy
---
src/backend/access/heap/heapam.c | 31 ++++++
src/backend/access/heap/pruneheap.c | 40 ++++++-
src/backend/commands/vacuumlazy.c | 161 +++++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 10 +-
src/include/access/heapam.h | 1 +
5 files changed, 234 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 72395a50b8..dd033668b6 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -9392,6 +9392,36 @@ heap_sync(Relation rel)
}
}
+/*
+ * Mask DEAD tuples at a HEAP page
+ *
+ * We introduce this function for wal_consistency_checking satisfaction at
+ * Hot Standby node.
+ *
+ * DEAD tuple on a master node has t_cid and t_infomask, which can not be
+ * obtained by WAL records applying on a Hot Standby node.
+ */
+static void
+mask_dead_tuples(Page page)
+{
+ OffsetNumber offnum;
+
+ for (offnum = FirstOffsetNumber; offnum <= PageGetMaxOffsetNumber(page); offnum = OffsetNumberNext(offnum))
+ {
+ ItemId lp = PageGetItemId(page, offnum);
+ char* htup_begin;
+
+ if (!ItemIdIsDead(lp))
+ continue;
+
+ if (!ItemIdHasStorage(lp))
+ continue;
+
+ htup_begin = (char *) PageGetItem(page, lp);
+ memset(htup_begin, MASK_MARKER, ItemIdGetLength(lp));
+ }
+}
+
/*
* Mask a heap page before performing consistency checks on it.
*/
@@ -9405,6 +9435,7 @@ heap_mask(char *pagedata, BlockNumber blkno)
mask_page_hint_bits(page);
mask_unused_space(page);
+ mask_dead_tuples(page);
for (off = 1; off <= PageGetMaxOffsetNumber(page); off++)
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c2f5343dac..ebbafe3275 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -43,6 +43,9 @@ typedef struct
bool marked[MaxHeapTuplesPerPage + 1];
} PruneState;
+/* Parameter for target deletion strategy in lazy vacuum */
+double target_index_deletion_factor = 0.01;
+
/* Local functions */
static int heap_prune_chain(Relation relation, Buffer buffer,
OffsetNumber rootoffnum,
@@ -405,7 +408,7 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (HeapTupleSatisfiesVacuum(&tup, OldestXmin, buffer)
== HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
{
- heap_prune_record_unused(prstate, rootoffnum);
+ heap_prune_record_dead(prstate, rootoffnum);
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
ndeleted++;
@@ -580,8 +583,10 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
{
- heap_prune_record_unused(prstate, chainitems[i]);
ndeleted++;
+ if (chainitems[i] == latestdead)
+ continue;
+ heap_prune_record_unused(prstate, chainitems[i]);
}
/*
@@ -598,9 +603,28 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
* redirect the root to the correct chain member.
*/
if (i >= nchain)
+ {
+ if (rootoffnum != latestdead)
+ {
+ if (ItemIdIsNormal(rootlp))
+ heap_prune_record_unused(prstate, latestdead);
+ else
+ {
+ /*
+ * We allow overlapping of redirected and dead items
+ */
+ heap_prune_record_redirect(prstate, rootoffnum, latestdead);
+ heap_prune_record_dead(prstate, latestdead);
+ }
+ }
heap_prune_record_dead(prstate, rootoffnum);
+ }
else
+ {
+ if (rootoffnum != latestdead)
+ heap_prune_record_unused(prstate, latestdead);
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
+ }
}
else if (nchain < 2 && ItemIdIsRedirected(rootlp))
{
@@ -653,7 +677,12 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
+
+ /*
+ * We suppress checking prstate->marked[offnum]. It is not the best idea,
+ * but this is most simplistic way to enable Dead Redirecting by
+ * overlapping Dead and Redirected states.
+ */
prstate->marked[offnum] = true;
}
@@ -706,7 +735,10 @@ heap_page_prune_execute(Buffer buffer,
OffsetNumber off = *offnum++;
ItemId lp = PageGetItemId(page, off);
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
}
/* Update all now-unused line pointers */
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 5649a70800..9a625b8dba 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -41,13 +41,16 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/nbtree.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
+#include "catalog/index.h"
#include "catalog/storage.h"
#include "commands/dbcommands.h"
#include "commands/progress.h"
#include "commands/vacuum.h"
+#include "executor/executor.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "portability/instr_time.h"
@@ -139,7 +142,6 @@ typedef struct LVRelStats
bool lock_waiter_detected;
} LVRelStats;
-
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
@@ -156,6 +158,8 @@ static void lazy_scan_heap(Relation onerel, int options,
bool aggressive);
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
+static void quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
@@ -734,10 +738,17 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
+ }
/*
* Report that we are now vacuuming the heap. We also increase
* the number of index scans here; note that by using
@@ -1379,10 +1390,16 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
- lazy_vacuum_index(Irel[i],
+ {
+ bool use_quick_strategy = (vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples < target_index_deletion_factor);
+
+ if (use_quick_strategy && (Irel[i]->rd_amroutine->amtargetdelete != NULL))
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
-
+ }
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
@@ -1671,6 +1688,142 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Get tuple from heap for a scan key building.
+ */
+static HeapTuple
+get_tuple_by_tid(Relation rel, ItemPointer tid)
+{
+ Buffer buffer;
+ Page page;
+ OffsetNumber offnum;
+ ItemId lp;
+ HeapTuple tuple;
+ bool needLock = !RELATION_IS_LOCAL(rel);
+ BlockNumber npages;
+
+ if (needLock)
+ LockRelationForExtension(rel, ExclusiveLock);
+ npages = RelationGetNumberOfBlocks(rel);
+ if (needLock)
+ UnlockRelationForExtension(rel, ExclusiveLock);
+ if (ItemPointerGetBlockNumber(tid) > npages)
+ return NULL;
+
+ buffer = ReadBufferExtended(rel, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid), RBM_NORMAL, NULL);
+ LockBuffer(buffer, BUFFER_LOCK_SHARE);
+
+ page = (Page) BufferGetPage(buffer);
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
+
+ /*
+ * VACUUM Races: someone already remove the tuple from HEAP. Ignore it.
+ */
+ if (!ItemIdIsUsed(lp))
+ {
+ UnlockReleaseBuffer(buffer);
+ return NULL;
+ }
+ /* Walk along the chain */
+ while (!ItemIdHasStorage(lp))
+ {
+ offnum = ItemIdGetRedirect(lp);
+ lp = PageGetItemId(page, offnum);
+ Assert(ItemIdIsUsed(lp));
+ }
+
+ /* Form a tuple */
+ tuple = palloc(sizeof(HeapTupleData));
+ ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
+ tuple->t_tableOid = RelationGetRelid(rel);
+ tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
+ tuple->t_len = ItemIdGetLength(lp);
+ UnlockReleaseBuffer(buffer);
+ return tuple;
+}
+
+/*
+ * quick_vacuum_index() -- quick vacuum one index relation.
+ *
+ * Delete all the index entries pointing to tuples listed in
+ * vacrelstats->dead_tuples.
+ */
+static void
+quick_vacuum_index(Relation irel, Relation hrel,
+ LVRelStats *vacrelstats)
+{
+ int tnum;
+ bool* found = palloc0(vacrelstats->num_dead_tuples*sizeof(bool));
+ IndexInfo* indexInfo = BuildIndexInfo(irel);
+ EState* estate = CreateExecutorState();
+ ExprContext* econtext = GetPerTupleExprContext(estate);
+ ExprState* predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
+ TupleTableSlot* slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+
+ IndexTargetDeleteResult stats;
+ IndexTargetDeleteInfo ivinfo;
+
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+
+ econtext->ecxt_scantuple = slot;
+
+ /* Get tuple from heap */
+ for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ {
+ HeapTuple tuple;
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+
+ /* Index entry for the TID was deleted early */
+ if (found[tnum])
+ continue;
+
+ /* Get a tuple from heap */
+ if ((tuple = get_tuple_by_tid(hrel, &(vacrelstats->dead_tuples[tnum]))) == NULL)
+ {
+ /*
+ * Tuple has 'not used' status.
+ */
+ found[tnum] = true;
+ continue;
+ }
+
+ /*
+ * Form values[] and isnull[] arrays from for index tuple
+ * by heap tuple
+ */
+ MemoryContextReset(econtext->ecxt_per_tuple_memory);
+
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * In a partial index, ignore tuples that don't satisfy the
+ * predicate.
+ */
+ if ((predicate != NULL) && (!ExecQual(predicate, econtext)))
+ {
+ found[tnum] = true;
+ continue;
+ }
+
+ FormIndexDatum(indexInfo, slot, estate, values, isnull);
+
+ /*
+ * Make attempt to delete some index entries by one tree descent.
+ * We use only a part of TID list, which contains not found TID's.
+ */
+ ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
+ ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples - tnum;
+ ivinfo.found_dead_tuples = found + tnum;
+ index_target_delete(&ivinfo, &stats, values, isnull);
+ }
+
+ pfree(found);
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a88ea6cfc9..e9e953ece6 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3243,7 +3243,15 @@ static struct config_real ConfigureNamesReal[] =
0.1, 0.0, 100.0,
NULL, NULL, NULL
},
-
+ {
+ {"target_index_deletion_factor", PGC_SIGHUP, AUTOVACUUM,
+ gettext_noop("Maximum number of vacuumed tuples as a fraction of reltuples where we can use target index vacuum strategy."),
+ NULL
+ },
+ &target_index_deletion_factor,
+ 0.01, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
{
{"checkpoint_completion_target", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval."),
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7497..c5a7bc8361 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -100,6 +100,7 @@ extern Relation heap_openrv_extended(const RangeVar *relation,
typedef struct HeapScanDescData *HeapScanDesc;
typedef struct ParallelHeapScanDescData *ParallelHeapScanDesc;
+extern double target_index_deletion_factor;
/*
* HeapScanIsValid
* True iff the heap scan is valid.
--
2.17.1
0004-v5-Retail-IndexTuple-Deletion-with-TID-sorting-in-Leaf.patch (text/x-patch, 6.7K) download
From ec49e3c00d21b5aa3ff9bca382cd8d745c5fe7f5 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Mon, 23 Jul 2018 09:38:45 +0500
Subject: [PATCH 4/4] v5 Retail IndexTuple Deletion with TID-sorting in Leaf
---
src/backend/access/nbtree/nbtree.c | 75 ++++++++++++++++++++----------
src/backend/commands/vacuumlazy.c | 8 ++--
src/include/access/genam.h | 2 +-
3 files changed, 55 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 44f4d0dc9f..50378e0f51 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -889,15 +889,19 @@ btbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
return stats;
}
-static int
-tid_list_search(ItemPointer tid, ItemPointer tid_list, int ntid)
-{
- for (int i = 0; i < ntid; i++)
- if (ItemPointerEquals(tid, &(tid_list[i])))
- return i;
- return -1;
-}
-
+/*
+ * Deletion of index entries pointing to heap tuples.
+ *
+ * Constraints:
+ * 1. TID list info->dead_tuples arranged in ASC order.
+ * 2. Logical duplicates of index tuples stored in DESC order.
+ *
+ * The function generates an insertion scan key and descent by btree for first
+ * index tuple what satisfies scan key and last TID in info->dead_tuples list.
+ * For the scan results it deletes all index entries, matched to the TID list.
+ *
+ * Result: a palloc'd struct containing statistical info.
+ */
IndexTargetDeleteResult*
bttargetdelete(IndexTargetDeleteInfo *info,
IndexTargetDeleteResult *stats,
@@ -916,33 +920,33 @@ bttargetdelete(IndexTargetDeleteInfo *info,
int ndeletable = 0;
OffsetNumber deletable[MaxOffsetNumber];
IndexTuple itup;
+ int pos = info->last_dead_tuple;
if (stats == NULL)
stats = (IndexTargetDeleteResult *) palloc0(sizeof(IndexTargetDeleteResult));
+ /* Assemble scankey */
itup = index_form_tuple(RelationGetDescr(irel), values, isnull);
skey = _bt_mkscankey(irel, itup);
/* Descend the tree and position ourselves on the target leaf page. */
- stack = _bt_search(irel, keysCount, skey, false, &buf, BT_READ, NULL);
- _bt_freestack(stack);
+ stack = _bt_search(irel, keysCount, skey, &info->dead_tuples[pos], false, &buf, BT_READ, NULL);
/* To prepare tuple entries search across index pages */
Assert(BufferIsValid(buf));
- offnum = _bt_binsrch(irel, buf, keysCount, skey, false);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], P_FIRSTDATAKEY(opaque), false);
for (;;)
{
int32 cmpval;
ItemId itemid;
IndexTuple itup;
- int pos;
/* Switch to the next page */
- if (P_IGNORE(opaque) || (offnum > PageGetMaxOffsetNumber(page)))
+ if (offnum > PageGetMaxOffsetNumber(page))
{
/*
* Before unlocking index page we need to delete
@@ -967,20 +971,27 @@ bttargetdelete(IndexTargetDeleteInfo *info,
break;
/*
- * Switch to the next index page
+ * Traverse to a next reliable index page
*/
- buf = _bt_relandgetbuf(irel, buf, opaque->btpo_next, BT_READ);
+ buf = _bt_moveright(irel, buf, keysCount, skey, &info->dead_tuples[pos],
+ false, true, stack, BT_READ, NULL);
page = BufferGetPage(buf);
_bt_checkpage(irel, buf);
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
- offnum = P_FIRSTDATAKEY(opaque);
- continue;
+ Assert(!P_IGNORE(opaque));
+ /* Set offnum to first potentially interesting item */
+ offnum = _bt_binsrch(irel, buf, keysCount, skey, &info->dead_tuples[pos], P_FIRSTDATAKEY(opaque), false);
+
+ if (offnum > PageGetMaxOffsetNumber(page))
+ break;
+ else
+ continue;
}
/*
* This index entry satisfied to the scan key?
*/
- cmpval = _bt_compare(irel, keysCount, skey, page, offnum);
+ cmpval = _bt_compare(irel, keysCount, skey, NULL, page, offnum);
if (cmpval != 0)
/* End of index entries, satisfied to the scan key */
@@ -992,15 +1003,28 @@ bttargetdelete(IndexTargetDeleteInfo *info,
*/
itemid = PageGetItemId(page, offnum);
itup = (IndexTuple) PageGetItem(page, itemid);
- pos = tid_list_search(&(itup->t_tid), info->dead_tuples, info->num_dead_tuples);
- if ((pos >= 0) && (!info->found_dead_tuples[pos]))
+ /*
+ * Search for next TID from presorted btree result comparable
+ * to TID from presorted dead_tuples tid list
+ */
+ while (pos >= 0)
{
- /* index entry for TID of dead tuple is found */
- deletable[ndeletable++] = offnum;
- info->found_dead_tuples[pos] = true;
+ int res = ItemPointerCompare(&(itup->t_tid), &info->dead_tuples[pos]);
+ if ((res == 0) && (!info->found_dead_tuples[pos]))
+ {
+ /* index entry for TID of dead tuple is found */
+ deletable[ndeletable++] = offnum;
+ info->found_dead_tuples[pos] = true;
+ }
+ else if (res > 0)
+ break;
+ pos--;
}
+ if (pos < 0)
+ break;
+
offnum = OffsetNumberNext(offnum);
}
@@ -1017,7 +1041,8 @@ bttargetdelete(IndexTargetDeleteInfo *info,
stats->tuples_removed += ndeletable;
}
- /* Release scan key, unpin and unlock buffer */
+ /* Release stack, scan key, unpin and unlock buffer */
+ _bt_freestack(stack);
_bt_freeskey(skey);
_bt_relbuf(irel, buf);
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 9a625b8dba..1b952fedd9 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -1770,7 +1770,7 @@ quick_vacuum_index(Relation irel, Relation hrel,
econtext->ecxt_scantuple = slot;
/* Get tuple from heap */
- for (tnum = 0; tnum < vacrelstats->num_dead_tuples; tnum++)
+ for (tnum = vacrelstats->num_dead_tuples-1; tnum >= 0; tnum--)
{
HeapTuple tuple;
Datum values[INDEX_MAX_KEYS];
@@ -1814,9 +1814,9 @@ quick_vacuum_index(Relation irel, Relation hrel,
* Make attempt to delete some index entries by one tree descent.
* We use only a part of TID list, which contains not found TID's.
*/
- ivinfo.dead_tuples = &(vacrelstats->dead_tuples[tnum]);
- ivinfo.num_dead_tuples = vacrelstats->num_dead_tuples - tnum;
- ivinfo.found_dead_tuples = found + tnum;
+ ivinfo.dead_tuples = vacrelstats->dead_tuples;
+ ivinfo.last_dead_tuple = tnum;
+ ivinfo.found_dead_tuples = found;
index_target_delete(&ivinfo, &stats, values, isnull);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 79b386eebd..297d357a7d 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -55,7 +55,7 @@ typedef struct IndexTargetDeleteInfo
{
Relation heapRelation;
Relation indexRelation; /* the index being vacuumed */
- int num_dead_tuples;
+ int last_dead_tuple;
ItemPointer dead_tuples;
bool* found_dead_tuples;
} IndexTargetDeleteInfo;
--
2.17.1
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-07-23 06:47 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2 siblings, 0 replies; 47+ messages in thread
From: Andrey Lepikhov @ 2018-07-23 06:47 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Юрий Соколов <funny.falcon@gmail.com>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
13.07.2018 00:00, Peter Geoghegan пишет:
> One area that might be worth investigating is retail index tuple
> deletion performed within the executor in the event of non-HOT
> updates.
I will try to use this idea. But now I developed a background worker
already (experimental state). It clean an index and heap relations with
a retail deletion approach.
It uses "soft" strategy with conditional buffer locks. Benchmarks on
pgbench test demonstrate same execution time as PostgreSQL without the
worker, but 90% of dead tuples in index and heap relations cleaned
without vacuum.
My main efforts is involved to tuning of the worker.
---
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-29 05:00 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2018-06-29 05:34 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2 siblings, 1 reply; 47+ messages in thread
From: Kuntal Ghosh @ 2018-06-29 05:00 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On Wed, Jun 27, 2018 at 12:10 PM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> I prepare third version of the patches. Summary:
> 1. Mask DEAD tuples at a page during consistency checking (See comments for
> the mask_dead_tuples() function).
- ItemIdSetDead(lp);
+ if (target_index_deletion_factor > 0)
+ ItemIdMarkDead(lp);
+ else
+ ItemIdSetDead(lp);
IIUC, you want to hold the storage of DEAD tuples to form the ScanKey
which is required for the index scan in the second phase of quick
vacuum strategy. To achieve that, you've only marked the tuple as DEAD
during pruning. Hence, PageRepairFragmentation cannot claim the space
for DEAD tuples since they still have storage associated with them.
But, you've WAL-logged it as DEAD tuples having no storage. So, when
the WAL record is replayed in standby(or crash recovery), the tuples
will be marked as DEAD having no storage and their space may be
reclaimed by PageRepairFragmentation. Now, if you do byte-by-byte
comparison with wal_consistency tool, it may fail even for normal
tuple as well. Please let me know if you feel the same way.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 05:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2018-06-29 05:34 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 06:39 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 1 reply; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-29 05:34 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: pgsql-hackers
On 29.06.2018 10:00, Kuntal Ghosh wrote:
> On Wed, Jun 27, 2018 at 12:10 PM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> I prepare third version of the patches. Summary:
>> 1. Mask DEAD tuples at a page during consistency checking (See comments for
>> the mask_dead_tuples() function).
>
> - ItemIdSetDead(lp);
> + if (target_index_deletion_factor > 0)
> + ItemIdMarkDead(lp);
> + else
> + ItemIdSetDead(lp);
> IIUC, you want to hold the storage of DEAD tuples to form the ScanKey
> which is required for the index scan in the second phase of quick
> vacuum strategy. To achieve that, you've only marked the tuple as DEAD
> during pruning. Hence, PageRepairFragmentation cannot claim the space
> for DEAD tuples since they still have storage associated with them.
> But, you've WAL-logged it as DEAD tuples having no storage. So, when
> the WAL record is replayed in standby(or crash recovery), the tuples
> will be marked as DEAD having no storage and their space may be
> reclaimed by PageRepairFragmentation. Now, if you do byte-by-byte
> comparison with wal_consistency tool, it may fail even for normal
> tuple as well. Please let me know if you feel the same way.
>
Thanks for your analysis.
In this development version of the patch I expect the same prune()
strategy on a master and standby (i.e. target_index_deletion_factor is
equal for both).
In this case storage of a DEAD tuple holds during replay or recovery in
the same way.
On some future step of development I plan to use more flexible prune()
strategy. This will require to append a 'isDeadStorageHold' field to the
WAL record.
--
Regards,
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 05:00 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2018-06-29 05:34 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-29 06:39 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 0 replies; 47+ messages in thread
From: Kuntal Ghosh @ 2018-06-29 06:39 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: pgsql-hackers
On Fri, Jun 29, 2018 at 11:04 AM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> On 29.06.2018 10:00, Kuntal Ghosh wrote:
>>
>> On Wed, Jun 27, 2018 at 12:10 PM, Andrey V. Lepikhov
>> <a.lepikhov@postgrespro.ru> wrote:
>>>
>>> I prepare third version of the patches. Summary:
>>> 1. Mask DEAD tuples at a page during consistency checking (See comments
>>> for
>>> the mask_dead_tuples() function).
>>
>>
>> - ItemIdSetDead(lp);
>> + if (target_index_deletion_factor > 0)
>> + ItemIdMarkDead(lp);
>> + else
>> + ItemIdSetDead(lp);
>> IIUC, you want to hold the storage of DEAD tuples to form the ScanKey
>> which is required for the index scan in the second phase of quick
>> vacuum strategy. To achieve that, you've only marked the tuple as DEAD
>> during pruning. Hence, PageRepairFragmentation cannot claim the space
>> for DEAD tuples since they still have storage associated with them.
>> But, you've WAL-logged it as DEAD tuples having no storage. So, when
>> the WAL record is replayed in standby(or crash recovery), the tuples
>> will be marked as DEAD having no storage and their space may be
>> reclaimed by PageRepairFragmentation. Now, if you do byte-by-byte
>> comparison with wal_consistency tool, it may fail even for normal
>> tuple as well. Please let me know if you feel the same way.
>>
> Thanks for your analysis.
> In this development version of the patch I expect the same prune() strategy
> on a master and standby (i.e. target_index_deletion_factor is equal for
> both).
> In this case storage of a DEAD tuple holds during replay or recovery in the
> same way.
Okay. That's acceptable for now.
> On some future step of development I plan to use more flexible prune()
> strategy. This will require to append a 'isDeadStorageHold' field to the WAL
> record.
That sounds interesting. I'll be waiting for your next patches.
Few minor comments:
+ qsort((void *)vacrelstats->dead_tuples,
vacrelstats->num_dead_tuples, sizeof(ItemPointerData),
tid_comparator);
+ ivinfo.isSorted = true;
My understanding is that vacrelstats->dead_tuples are already sorted
based on their tids. I'm referring to the following part in
lazy_scan_heap(),
for (blk=0 to nblocks)
{
for (offset=1 to max offset)
{
if certain conditions are met store the tuple in
vacrelstats->dead_tuples using lazy_record_dead_tuple();
}
}
So, you don't have to sort them again, right?
+ slot = MakeSingleTupleTableSlot(RelationGetDescr(hrel));
+ econtext->ecxt_scantuple = slot;
+ ExecDropSingleTupleTableSlot(slot);
You don't have to do this for every tuple. Before storing the tuple,
you can just call MemoryContextReset(econtext->ecxt_per_tuple_memory);
Perhaps, you can check IndexBuildHeapRangeScan for the details.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-27 06:40 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-29 09:03 ` Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 0 replies; 47+ messages in thread
From: Dilip Kumar @ 2018-06-29 09:03 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; pgsql-hackers; Masahiko Sawada <sawada.mshk@gmail.com>
On Wed, Jun 27, 2018 at 12:10 PM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
>
>
> On 23.06.2018 00:43, Peter Geoghegan wrote:
>>
>> On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov
>> <a.lepikhov@postgrespro.ru> wrote:
>>>
>>> According to your feedback, i develop second version of the patch.
>>> In this version:
I had quick look at your patch, I have few comments.
1.
+ if (use_quick_strategy)
+ quick_vacuum_index(Irel[i], onerel, vacrelstats);
+ else
+ lazy_vacuum_index(Irel[i],
I noticed that inside quick_vacuum_index again we check if we don't
have am routine for the target
delete then we call lazy_vacuum_index. I think we can have that check
here itself?
2.
+ else
+ {
+ IndexBulkDeleteResult **stats;
+
+ lazy_vacuum_index(irel, stats, vacrelstats);
+ }
These stats will be lost, I think if you fix first then this will
issue will be solved, otherwise, you might
need to pass indexstats as a parameter to quick_vacuum_index function.
3.
+}
+
+#include "access/nbtree.h"
+#include "catalog/index.h"
+#include "executor/executor.h"
You can move these header inclusions at the top of the file.
4. Many places PG coding standard is not followed, few examples
a.
+ bool use_quick_strategy =
(vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples <
target_index_deletion_factor);
+
space between operator and variable
vacrelstats->num_dead_tuples/vacrelstats->old_live_tuples ->
vacrelstats->num_dead_tuples / vacrelstats->old_live_tuples
b.qsort((void *)vacrelstats -> qsort((void *) vacrelstats
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
@ 2018-06-26 10:31 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-27 04:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-27 23:35 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
1 sibling, 2 replies; 47+ messages in thread
From: Masahiko Sawada @ 2018-06-26 10:31 UTC (permalink / raw)
To: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; pgsql-hackers
On Fri, Jun 22, 2018 at 8:24 PM, Andrey V. Lepikhov
<a.lepikhov@postgrespro.ru> wrote:
> Hi,
> According to your feedback, i develop second version of the patch.
> In this version:
> 1. High-level functions index_beginscan(), index_rescan() not used. Tree
> descent made by _bt_search(). _bt_binsrch() used for positioning on the
> page.
> 2. TID list introduced in amtargetdelete() interface. Now only one tree
> descent needed for deletion all tid's from the list with equal scan key
> value - logical duplicates deletion problem.
> 3. Only one WAL record for index tuple deletion per leaf page per
> amtargetdelete() call.
> 4. VACUUM can sort TID list preliminary for more quick search of duplicates.
>
> Background worker will come later.
>
>
Thank you for updating patches! Here is some comments for the latest patch.
+static void
+quick_vacuum_index(Relation irel, Relation hrel,
+ IndexBulkDeleteResult **overall_stats,
+ LVRelStats *vacrelstats)
+{
(snip)
+ /*
+ * Collect statistical info
+ */
+ lazy_cleanup_index(irel, *overall_stats, vacrelstats);
+}
I think that we should not call lazy_cleanup_index at the end of
quick_vacuum_index because we call it multiple times during a lazy
vacuum and index statistics can be changed during vacuum. We already
call lazy_cleanup_index at the end of lazy_scan_heap.
bttargetdelete doesn't delete btree pages even if pages become empty.
I think we should do that. Otherwise empty page never be recycled. But
please note that if we delete btree pages during bttargetdelete,
recyclable pages might not be recycled. That is, if we choose the
target deletion method every time then the deleted-but-not-recycled
pages could never be touched, unless reaching
vacuum_cleanup_index_scale_factor. So I think we need to either run
bulk-deletion method or do cleanup index before btpo.xact wraparound.
+ ivinfo.indexRelation = irel;
+ ivinfo.heapRelation = hrel;
+ qsort((void *)vacrelstats->dead_tuples,
vacrelstats->num_dead_tuples, sizeof(ItemPointerData),
tid_comparator);
I think the sorting vacrelstats->dead_tuples is not necessary because
garbage TIDs are stored in a sorted order.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-26 10:31 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-06-27 04:05 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
1 sibling, 0 replies; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-27 04:05 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; pgsql-hackers
On 26.06.2018 15:31, Masahiko Sawada wrote:
> On Fri, Jun 22, 2018 at 8:24 PM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> Hi,
>> According to your feedback, i develop second version of the patch.
>> In this version:
>> 1. High-level functions index_beginscan(), index_rescan() not used. Tree
>> descent made by _bt_search(). _bt_binsrch() used for positioning on the
>> page.
>> 2. TID list introduced in amtargetdelete() interface. Now only one tree
>> descent needed for deletion all tid's from the list with equal scan key
>> value - logical duplicates deletion problem.
>> 3. Only one WAL record for index tuple deletion per leaf page per
>> amtargetdelete() call.
>> 4. VACUUM can sort TID list preliminary for more quick search of duplicates.
>>
>> Background worker will come later.
>>
>>
>
> Thank you for updating patches! Here is some comments for the latest patch.
>
> +static void
> +quick_vacuum_index(Relation irel, Relation hrel,
> + IndexBulkDeleteResult **overall_stats,
> + LVRelStats *vacrelstats)
> +{
> (snip)
> + /*
> + * Collect statistical info
> + */
> + lazy_cleanup_index(irel, *overall_stats, vacrelstats);
> +}
>
> I think that we should not call lazy_cleanup_index at the end of
> quick_vacuum_index because we call it multiple times during a lazy
> vacuum and index statistics can be changed during vacuum. We already
> call lazy_cleanup_index at the end of lazy_scan_heap.
>
Ok
> bttargetdelete doesn't delete btree pages even if pages become empty.
> I think we should do that. Otherwise empty page never be recycled. But
> please note that if we delete btree pages during bttargetdelete,
> recyclable pages might not be recycled. That is, if we choose the
> target deletion method every time then the deleted-but-not-recycled
> pages could never be touched, unless reaching
> vacuum_cleanup_index_scale_factor. So I think we need to either run
> bulk-deletion method or do cleanup index before btpo.xact wraparound.
>
> + ivinfo.indexRelation = irel;
> + ivinfo.heapRelation = hrel;
> + qsort((void *)vacrelstats->dead_tuples,
> vacrelstats->num_dead_tuples, sizeof(ItemPointerData),
> tid_comparator);
>
Ok. I think caller of bttargetdelete() must decide when to make index
cleanup.
> I think the sorting vacrelstats->dead_tuples is not necessary because
> garbage TIDs are stored in a sorted order.
>
Sorting was introduced because I keep in mind background worker and more
flexible cleaning strategies, not only full tuple-by-tuple relation and
block scan.
Caller of bttargetdelete() can set info->isSorted to prevent sorting
operation.
> Regards,
>
> --
> Masahiko Sawada
> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
> NTT Open Source Software Center
>
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-26 10:31 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Masahiko Sawada <sawada.mshk@gmail.com>
@ 2018-06-27 23:35 ` Peter Geoghegan <pg@bowt.ie>
1 sibling, 0 replies; 47+ messages in thread
From: Peter Geoghegan @ 2018-06-27 23:35 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>; pgsql-hackers
On Tue, Jun 26, 2018 at 3:31 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> bttargetdelete doesn't delete btree pages even if pages become empty.
> I think we should do that. Otherwise empty page never be recycled. But
> please note that if we delete btree pages during bttargetdelete,
> recyclable pages might not be recycled. That is, if we choose the
> target deletion method every time then the deleted-but-not-recycled
> pages could never be touched, unless reaching
> vacuum_cleanup_index_scale_factor. So I think we need to either run
> bulk-deletion method or do cleanup index before btpo.xact wraparound.
As you pointed out, we can certainly never fully delete or recycle
half-dead pages using bttargetdelete. We already need to make some
kind of compromise around page deletion, and it may not be necessary
to insist that bttargetdelete does any kind of page deletion. I'm
unsure of that, though.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: [WIP] [B-Tree] Retail IndexTuple deletion
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 21:54 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Re: [WIP] [B-Tree] Retail IndexTuple deletion Peter Geoghegan <pg@bowt.ie>
@ 2018-06-20 04:04 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
3 siblings, 0 replies; 47+ messages in thread
From: Andrey V. Lepikhov @ 2018-06-20 04:04 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-hackers
On 19.06.2018 04:05, Peter Geoghegan wrote:
> On Mon, Jun 18, 2018 at 2:54 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov
>> <a.lepikhov@postgrespro.ru> wrote:
>>> Patch '0001-retail-indextuple-deletion' introduce new function
>>> amtargetdelete() in access method interface. Patch
>>> '0002-quick-vacuum-strategy' implements this function for an alternative
>>> strategy of lazy index vacuum, called 'Quick Vacuum'.
>>
>> My compiler shows the following warnings:
>
> Some real feedback:
>
> What we probably want to end up with here is new lazyvacuum.c code
> that does processing for one heap page (and associated indexes) that
> is really just a "partial" lazy vacuum. Though it won't do things like
> advance relfrozenxid, it will do pruning for the heap page, index
> tuple killing, and finally heap tuple killing. It will do all of these
> things reliably, just like traditional lazy vacuum. This will be what
> your background worker eventually uses.
>
It is final goal of the patch.
> I doubt that you can use routines like index_beginscan() within
> bttargetdelete() at all. I think that you need something closer to
> _bt_doinsert() or _bt_pagedel(), that manages its own scan (your code
> should probably live in nbtpage.c). It does not make sense to teach
> external, generic routines like index_beginscan() about heap TID being
> an implicit final index attribute, which is one reason for this (I'm
> assuming that this patch relies on my own patch). Another reason is
> that you need to hold an exclusive buffer lock at the point that you
> identify the tuple to be killed, until the point that you actually
> kill it. You don't do that now.
>
> IOW, the approach you've taken in bttargetdelete() isn't quite correct
> because you imagine that it's okay to occasionally "lose" the index
> tuple that you originally found, and just move on. That needs to be
> 100% reliable, or else we'll end up with index tuples that point to
> the wrong heap tuples in rare cases with concurrent insertions. As I
> said, we want a "partial" lazy vacuum here, which must mean that it's
> reliable. Note that _bt_pagedel() actually calls _bt_search() when it
> deletes a page. Your patch will not be the first patch that makes
> nbtree vacuuming do an index scan. You should be managing your own
> insertion scan key, much like _bt_pagedel() does. If you use my patch,
> _bt_search() can be taught to look for a specific heap TID.
>
Agree with this notes. Corrections will made in the next version of the
patch.
> Finally, doing things this way would let you delete multiple
> duplicates in one shot, as I described in an earlier e-mail. Only a
> single descent of the tree is needed to delete quite a few index
> tuples, provided that they all happen to be logical duplicates. Again,
> your background worker will take advantage of this.
>
It is very interesting idea. According to this, I plan to change
bttargetdelete() interface as follows:
IndexTargetDeleteStats*
amtargetdelete(IndexTargetDeleteInfo *info,
IndexTargetDeleteStats *stats,
Datum *values, bool *isnull);
where structure IndexTargetDeleteInfo contains a TID list of dead heap
tuples. All index entries, corresponding to this list, may be deleted
(or only some of it) by one call of amtargetdelete() function with
single descent of the tree.
> This code does not follow the Postgres style:
>
>> - else
>> + }
>> + else {
>> + if (rootoffnum != latestdead)
>> + heap_prune_record_unused(prstate, latestdead);
>> heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
>> + }
>> }
>
> Please be more careful about that. I find it very distracting.
>
Done
--
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 47+ messages in thread
end of thread, other threads:[~2019-01-31 11:20 UTC | newest]
Thread overview: 47+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-06-18 04:39 [WIP] [B-Tree] Retail IndexTuple deletion Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-18 19:59 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 20:42 ` Claudio Freire <klaussfreire@gmail.com>
2018-06-18 21:15 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 21:54 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:05 ` Peter Geoghegan <pg@bowt.ie>
2018-06-18 23:33 ` Peter Geoghegan <pg@bowt.ie>
2018-06-19 02:29 ` Peter Geoghegan <pg@bowt.ie>
2018-06-19 09:33 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-19 17:38 ` Peter Geoghegan <pg@bowt.ie>
2018-06-22 11:24 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-22 19:43 ` Peter Geoghegan <pg@bowt.ie>
2018-06-22 20:14 ` Peter Geoghegan <pg@bowt.ie>
2018-06-25 18:20 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-27 06:40 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-28 00:00 ` Peter Geoghegan <pg@bowt.ie>
2018-06-28 05:36 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 09:07 ` Юрий Соколов <funny.falcon@gmail.com>
2018-07-02 14:29 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-02 16:28 ` Peter Geoghegan <pg@bowt.ie>
2018-07-02 19:40 ` Peter Geoghegan <pg@bowt.ie>
2018-07-03 12:17 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-12 19:00 ` Peter Geoghegan <pg@bowt.ie>
2018-07-19 11:29 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-07-20 18:11 ` Peter Geoghegan <pg@bowt.ie>
2018-07-24 11:38 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-08-07 07:19 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-09 00:19 ` Peter Geoghegan <pg@bowt.ie>
2018-08-09 03:43 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-08-15 07:17 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-08-15 09:28 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-05 12:25 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-06 10:26 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-20 04:35 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-09-21 03:52 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-11-29 13:27 ` Dmitry Dolgov <9erthalion6@gmail.com>
2019-01-31 11:20 ` Andres Freund <andres@anarazel.de>
2018-07-23 06:25 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-07-23 06:47 ` Andrey Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 05:00 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2018-06-29 05:34 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-29 06:39 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2018-06-29 09:03 ` Dilip Kumar <dilipbalaut@gmail.com>
2018-06-26 10:31 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-06-27 04:05 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
2018-06-27 23:35 ` Peter Geoghegan <pg@bowt.ie>
2018-06-20 04:04 ` Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox