public inbox for [email protected]
help / color / mirror / Atom feed[PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
45+ messages / 17 participants
[nested] [flat]
* [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
@ 2021-02-08 10:46 Pavel Borisov <[email protected]>
2021-02-08 19:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-03 03:10 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Geoghegan <[email protected]>
0 siblings, 3 replies; 45+ messages in thread
From: Pavel Borisov @ 2021-02-08 10:46 UTC (permalink / raw)
To: Postgres hackers <[email protected]>
Hi, hackers!
It seems that if btree index with a unique constraint is corrupted by
duplicates, amcheck now can not catch this. Reindex becomes impossible as
it throws an error but otherwise the index will let the user know that it
is corrupted, and amcheck will tell that the index is clean. So I'd like to
propose a short patch to improve amcheck for checking the unique
constraint. It will output tid's of tuples that are duplicated in the index
(i.e. more than one tid for the same index key is visible) and the user can
easily investigate and delete corresponding table entries.
0001 - is the actual patch, and
0002 - is a temporary hack for testing. It will allow inserting duplicates
in a table even if an index with the exact name "idx" has a unique
constraint (generally it is prohibited to insert). Then a new amcheck will
tell us about these duplicates. It's pity but testing can not be done
automatically, as it needs a core recompile. For testing I'd recommend a
protocol similar to the following:
- Apply patch 0002
- Set autovaccum = off in postgresql.conf
*create table tbl2 (a varchar(50), b varchar(150), c bytea, d
varchar(50));create unique index idx on tbl2(a,b);insert into tbl2 select
i::text::varchar, i::text::varchar, i::text::bytea, i::text::varchar from
generate_series(0,3) as i, generate_series(0,10000) as x;*
So we'll have a generous amount of duplicates in the table and index. Then:
*create extension amcheck;*
*select bt_index_check('idx', true);*
There will be output about each pair of duplicates: index tid's, position
in a posting list (if the index item is deduplicated) and table tid's.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 218 and posting 219 (point to heap tid=(126,93) and tid=(126,94))
page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 219 and posting 220 (point to heap tid=(126,94) and tid=(126,95))
page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 220 and posting 221 (point to heap tid=(126,95) and tid=(126,96))
page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 221 and tid=(26,7) posting 0 (point to heap tid=(126,96) and
tid=(126,97)) page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
posting 0 and posting 1 (point to heap tid=(126,97) and tid=(126,98)) page
lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
posting 1 and posting 2 (point to heap tid=(126,98) and tid=(126,99)) page
lsn=0/1B3D420.
Things to notice in the test output:
- cross-page duplicates (when some of them on the one index page and the
other are on the next). (Under patch 0002 they are marked by an additional
message "*INFO: cross page equal keys"* to catch them among the other)
- If we delete table entries corresponding to some duplicates in between
the other duplicates (vacuum should be off), the message for this entry
should disappear but the other duplicates should be detected as previous.
*delete from tbl2 where ctid::text='(126,94)';*
*select bt_index_check('idx', true);*
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 218 and posting 220 (point to heap tid=(126,93) and tid=(126,95))
page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 220 and posting 221 (point to heap tid=(126,95) and tid=(126,96))
page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
posting 221 and tid=(26,7) posting 0 (point to heap tid=(126,96) and
tid=(126,97)) page lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
posting 0 and posting 1 (point to heap tid=(126,97) and tid=(126,98)) page
lsn=0/1B3D420.
WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
posting 1 and posting 2 (point to heap tid=(126,98) and tid=(126,99)) page
lsn=0/1B3D420.
Caveat: if the first entry on the next index page has a key equal to the
key on a previous page AND all heap tid's corresponding to this entry are
invisible, currently cross-page check can not detect unique constraint
violation between previous index page entry and 2nd, 3d and next current
index page entries. In this case, there would be a message that recommends
doing VACUUM to remove the invisible entries from the index and repeat the
check. (Generally, it is recommended to do vacuum before the check, but for
the testing purpose I'd recommend turning it off to check the detection of
visible-invisible-visible duplicates scenarios)
Your feedback is very much welcome, as usual.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v1-0002-Ignore-unique-constraint-check-on-btree-insert.-P.patch (2.7K, ../../CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg@mail.gmail.com/3-v1-0002-Ignore-unique-constraint-check-on-btree-insert.-P.patch)
download | inline diff:
From a54ab438e911773f18409b2e2b951dc518d715e1 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 8 Feb 2021 11:57:49 +0400
Subject: [PATCH v1 2/2] Ignore unique constraint check on btree insert. Patch
for testing unique constraint checking in amcheck. Used to construct btree
index with violated unique constraint which is not generally possible.
Also increase log level of some debug messages in btree amcheck.
XXX Not for merge, testing only !!!!
---
contrib/amcheck/verify_nbtree.c | 4 ++--
src/backend/access/nbtree/nbtinsert.c | 16 ++++++++++++----
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 8a5809f017e..b4f012f5214 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -1729,7 +1729,7 @@ bt_target_page_check(BtreeCheckState *state)
*/
if (state->indexinfo->ii_Unique && rightkey && P_ISLEAF(topaque))
{
- elog(DEBUG2, "check cross page unique condition");
+ elog(INFO, "check cross page unique condition");
/*
* Make _bt_compare compare only index keys without heap TIDs.
@@ -1741,7 +1741,7 @@ bt_target_page_check(BtreeCheckState *state)
/* First key on next page is same */
if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
{
- elog(DEBUG2, "cross page equal keys");
+ elog(INFO, "cross page equal keys");
state->target = palloc_btree_page(state,
state->targetblock + 1);
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c
index e3336039125..ffbbf5c63e0 100644
--- a/src/backend/access/nbtree/nbtinsert.c
+++ b/src/backend/access/nbtree/nbtinsert.c
@@ -570,10 +570,15 @@ _bt_check_unique(Relation rel, BTInsertState insertstate, Relation heapRel,
*/
if (checkUnique == UNIQUE_CHECK_PARTIAL)
{
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- *is_unique = false;
- return InvalidTransactionId;
+ /* FIXME Ugliest crutch to test amcheck */
+ if (strcmp(RelationGetRelationName(rel), "idx") != 0)
+ {
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+
+ *is_unique = false;
+ return InvalidTransactionId;
+ }
}
/*
@@ -616,6 +621,9 @@ _bt_check_unique(Relation rel, BTInsertState insertstate, Relation heapRel,
SnapshotSelf, NULL))
{
/* Normal case --- it's still live */
+ /* FIXME Ugliest crutch to test amcheck */
+ if (strcmp(RelationGetRelationName(rel), "idx") == 0)
+ break;
}
else
{
--
2.28.0
[application/octet-stream] v1-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch (13.1K, ../../CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg@mail.gmail.com/4-v1-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch)
download | inline diff:
From 68d7fc042e3611ef70e90df057c3dee6ed513727 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 8 Feb 2021 12:26:08 +0400
Subject: [PATCH v1 1/2] Make amcheck checking UNIQUE constraint for btree
index. On index with unique constraint check that only one table entry
for the equal keys (including all posting list entries) is visible. Report
error if not and show all index entries violating the constraint under
warning level.
Authors: Anastasia Lubennikova <[email protected]>, Pavel Borisov <[email protected]>
---
contrib/amcheck/verify_nbtree.c | 267 +++++++++++++++++++++++++++++++-
1 file changed, 263 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index b8c7793d9e0..8a5809f017e 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -83,6 +83,13 @@ typedef struct BtreeCheckState
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking.
+ * Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -148,8 +155,20 @@ static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -187,6 +206,7 @@ static ItemId PageGetItemIdCareful(BtreeCheckState *state, BlockNumber block,
static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
IndexTuple itup, bool nonpivot);
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
+static bool errflag; /* Output ERROR at the end of amcheck */
/*
* bt_index_check(index regclass, heapallindexed boolean)
@@ -449,6 +469,15 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->indexinfo = BuildIndexInfo(state->rel);
+ /*
+ * We need a snapshot it to check uniqueness of the index
+ * For better performance, take it once per index check.
+ */
+ if (state->indexinfo->ii_Unique)
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ else
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -632,7 +661,16 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
+
+ if (errflag == true)
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index \"%s\" is corrupted. There are tuples violating UNIQUE constraint",
+ RelationGetRelationName(state->rel)),
+ errdetail_internal("Details are in the previous log messages under WARNING priority")));
}
/*
@@ -1006,6 +1044,142 @@ bt_recheck_sibling_links(BtreeCheckState *state,
btpo_prev_from_target)));
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ errflag = true;
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) "
+ "page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ (uint32) (state->targetlsn >> 32),
+ (uint32) state->targetlsn)));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, state->targetblock,
+ offset, i);
+ }
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = state->targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list.
+ * If TID is visible, save info about it for next comparisons in the loop in
+ * bt_page_check(). If also lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, state->targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = state->targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != state->targetblock)
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness may be violated for index \"%s\": "
+ "First key on an index page %u is equal to the key on the "
+ "previous page %u and is invisible. Cross-page unique "
+ "constraint violation can be missed. Vacuum the table "
+ "and repeat the check.",
+ RelationGetRelationName(state->rel),
+ state->targetblock, *lVis_block)));
+}
+
/*
* Function performs the following checks on target page, or pages ancillary to
* target page:
@@ -1026,6 +1200,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1047,6 +1224,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber offset;
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for posting
+ * tuple. for non-posting tuple (-1)
+ */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1446,6 +1630,39 @@ bt_target_page_check(BtreeCheckState *state)
(uint32) state->targetlsn)));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking
+ * heap tuples visibility.
+ */
+ if (state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+ /* Invalidate scankey tid to make _bt_compare compare only keys
+ * in the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1463,12 +1680,14 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1503,6 +1722,43 @@ bt_target_page_check(BtreeCheckState *state)
(uint32) (state->targetlsn >> 32),
(uint32) state->targetlsn)));
}
+
+ /*
+ * If index has unique constraint check that not more than one found
+ * equal items is visible.
+ */
+ if (state->indexinfo->ii_Unique && rightkey && P_ISLEAF(topaque))
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ state->targetblock + 1);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, state->targetblock + 1,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1548,9 +1804,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1713,6 +1971,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
--
2.28.0
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-02-08 19:31 ` Pavel Borisov <[email protected]>
2 siblings, 0 replies; 45+ messages in thread
From: Pavel Borisov @ 2021-02-08 19:31 UTC (permalink / raw)
To: Postgres hackers <[email protected]>
On Mon, 8 Feb 2021, 14:46 Pavel Borisov <[email protected] wrote:
> Hi, hackers!
>
> It seems that if btree index with a unique constraint is corrupted by
> duplicates, amcheck now can not catch this. Reindex becomes impossible as
> it throws an error but otherwise the index will let the user know that it
> is corrupted, and amcheck will tell that the index is clean. So I'd like to
> propose a short patch to improve amcheck for checking the unique
> constraint. It will output tid's of tuples that are duplicated in the index
> (i.e. more than one tid for the same index key is visible) and the user can
> easily investigate and delete corresponding table entries.
>
> 0001 - is the actual patch, and
> 0002 - is a temporary hack for testing. It will allow inserting duplicates
> in a table even if an index with the exact name "idx" has a unique
> constraint (generally it is prohibited to insert). Then a new amcheck will
> tell us about these duplicates. It's pity but testing can not be done
> automatically, as it needs a core recompile. For testing I'd recommend a
> protocol similar to the following:
>
> - Apply patch 0002
> - Set autovaccum = off in postgresql.conf
>
>
>
> *create table tbl2 (a varchar(50), b varchar(150), c bytea, d
> varchar(50));create unique index idx on tbl2(a,b);insert into tbl2 select
> i::text::varchar, i::text::varchar, i::text::bytea, i::text::varchar from
> generate_series(0,3) as i, generate_series(0,10000) as x;*
>
> So we'll have a generous amount of duplicates in the table and index. Then:
>
> *create extension amcheck;*
> *select bt_index_check('idx', true);*
>
> There will be output about each pair of duplicates: index tid's, position
> in a posting list (if the index item is deduplicated) and table tid's.
>
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 218 and posting 219 (point to heap tid=(126,93) and tid=(126,94))
> page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 219 and posting 220 (point to heap tid=(126,94) and tid=(126,95))
> page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 220 and posting 221 (point to heap tid=(126,95) and tid=(126,96))
> page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 221 and tid=(26,7) posting 0 (point to heap tid=(126,96) and
> tid=(126,97)) page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
> posting 0 and posting 1 (point to heap tid=(126,97) and tid=(126,98)) page
> lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
> posting 1 and posting 2 (point to heap tid=(126,98) and tid=(126,99)) page
> lsn=0/1B3D420.
>
> Things to notice in the test output:
> - cross-page duplicates (when some of them on the one index page and the
> other are on the next). (Under patch 0002 they are marked by an additional
> message "*INFO: cross page equal keys"* to catch them among the other)
>
> - If we delete table entries corresponding to some duplicates in between
> the other duplicates (vacuum should be off), the message for this entry
> should disappear but the other duplicates should be detected as previous.
>
> *delete from tbl2 where ctid::text='(126,94)';*
> *select bt_index_check('idx', true);*
>
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 218 and posting 220 (point to heap tid=(126,93) and tid=(126,95))
> page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 220 and posting 221 (point to heap tid=(126,95) and tid=(126,96))
> page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,6)
> posting 221 and tid=(26,7) posting 0 (point to heap tid=(126,96) and
> tid=(126,97)) page lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
> posting 0 and posting 1 (point to heap tid=(126,97) and tid=(126,98)) page
> lsn=0/1B3D420.
> WARNING: index uniqueness is violated for index "idx": Index tid=(26,7)
> posting 1 and posting 2 (point to heap tid=(126,98) and tid=(126,99)) page
> lsn=0/1B3D420.
>
> Caveat: if the first entry on the next index page has a key equal to the
> key on a previous page AND all heap tid's corresponding to this entry are
> invisible, currently cross-page check can not detect unique constraint
> violation between previous index page entry and 2nd, 3d and next current
> index page entries. In this case, there would be a message that recommends
> doing VACUUM to remove the invisible entries from the index and repeat the
> check. (Generally, it is recommended to do vacuum before the check, but for
> the testing purpose I'd recommend turning it off to check the detection of
> visible-invisible-visible duplicates scenarios)
>
> Your feedback is very much welcome, as usual.
>
> --
> Best regards,
> Pavel Borisov
>
> Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
>
There was typo, I mean, initially"...index will NOT let the user know that
it is corrupted..."
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-02-08 21:46 ` Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2 siblings, 1 reply; 45+ messages in thread
From: Mark Dilger @ 2021-02-08 21:46 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; Anastasia Lubennikova <[email protected]>; +Cc: Postgres hackers <[email protected]>
> On Feb 8, 2021, at 2:46 AM, Pavel Borisov <[email protected]> wrote:
>
> 0002 - is a temporary hack for testing. It will allow inserting duplicates in a table even if an index with the exact name "idx" has a unique constraint (generally it is prohibited to insert). Then a new amcheck will tell us about these duplicates. It's pity but testing can not be done automatically, as it needs a core recompile. For testing I'd recommend a protocol similar to the following:
>
> - Apply patch 0002
> - Set autovaccum = off in postgresql.conf
Thanks Pavel and Anastasia for working on this!
Updating pg_catalog directly is ugly, but the following seems a simpler way to set up a regression test than having to recompile. What do you think?
CREATE TABLE junk (t text);
CREATE UNIQUE INDEX junk_idx ON junk USING btree (t);
INSERT INTO junk (t) VALUES ('fee'), ('fi'), ('fo'), ('fum');
UPDATE pg_catalog.pg_index
SET indisunique = false
WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'junk');
INSERT INTO junk (t) VALUES ('fee'), ('fi'), ('fo'), ('fum');
UPDATE pg_catalog.pg_index
SET indisunique = true
WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'junk');
SELECT * FROM junk;
t
-----
fee
fi
fo
fum
fee
fi
fo
fum
(8 rows)
\d junk
Table "public.junk"
Column | Type | Collation | Nullable | Default
--------+------+-----------+----------+---------
t | text | | |
Indexes:
"junk_idx" UNIQUE, btree (t)
\d junk_idx
Index "public.junk_idx"
Column | Type | Key? | Definition
--------+------+------+------------
t | text | yes | t
unique, btree, for table "public.junk"
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-02-09 18:43 ` Pavel Borisov <[email protected]>
2021-02-09 18:57 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-09 19:41 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Zhihong Yu <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
0 siblings, 3 replies; 45+ messages in thread
From: Pavel Borisov @ 2021-02-09 18:43 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>
вт, 9 февр. 2021 г. в 01:46, Mark Dilger <[email protected]>:
>
>
> > On Feb 8, 2021, at 2:46 AM, Pavel Borisov <[email protected]>
> wrote:
> >
> > 0002 - is a temporary hack for testing. It will allow inserting
> duplicates in a table even if an index with the exact name "idx" has a
> unique constraint (generally it is prohibited to insert). Then a new
> amcheck will tell us about these duplicates. It's pity but testing can not
> be done automatically, as it needs a core recompile. For testing I'd
> recommend a protocol similar to the following:
> >
> > - Apply patch 0002
> > - Set autovaccum = off in postgresql.conf
>
> Thanks Pavel and Anastasia for working on this!
>
> Updating pg_catalog directly is ugly, but the following seems a simpler
> way to set up a regression test than having to recompile. What do you
> think?
>
> Very nice idea, thanks!
I've made a regression test based on it. PFA v.2 of a patch. Now it doesn't
need anything external for testing.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v2-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch (36.3K, ../../CALT9ZEFYsYPms0SPp4ZxpFCOBQbgE+iopoVKzU8Pni8Y6+RKtQ@mail.gmail.com/3-v2-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch)
download | inline diff:
From abaa141811273ebcc511218ec3f40713717a282c Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 8 Feb 2021 12:26:08 +0400
Subject: [PATCH v2] Make amcheck checking UNIQUE constraint for btree index.
On index with unique constraint ake check that only one table entry for the
equal keys (including all posting list entries) is visible. Report error if
not and show all index entries violating the constraint under warning level.
Authors: Anastasia Lubennikova <[email protected]>, Pavel Borisov <[email protected]>
---
contrib/amcheck/expected/check_btree.out | 138 +++++++++++
contrib/amcheck/sql/check_btree.sql | 24 ++
contrib/amcheck/verify_nbtree.c | 278 ++++++++++++++++++++++-
3 files changed, 436 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 13848b7449b..875ac1355ae 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,11 +177,149 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+SELECT bt_index_check('bttest_unique_idx', true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 2 (point to heap tid=(0,1) and tid=(0,3)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,3) and tid=(0,4)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and posting 4 (point to heap tid=(0,4) and tid=(0,5)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 4 and tid=(1,3) posting 0 (point to heap tid=(0,5) and tid=(0,6)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(0,6) and tid=(1,1)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,1) and tid=(1,2)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,2) and tid=(1,3)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,3) and tid=(1,4)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and tid=(1,4) posting 0 (point to heap tid=(1,4) and tid=(1,5)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(1,5) and tid=(1,6)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(1,6) and tid=(2,1)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,1) and tid=(2,2)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,2) and tid=(2,3)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and tid=(1,5) posting 0 (point to heap tid=(2,3) and tid=(2,4)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(2,4) and tid=(2,5)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(2,5) and tid=(2,6)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(2,6) and tid=(3,1)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,1) and tid=(3,2)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and tid=(1,6) posting 0 (point to heap tid=(3,2) and tid=(3,3)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(3,3) and tid=(3,4)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(3,4) and tid=(3,5)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(3,5) and tid=(3,6)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(3,6) and tid=(4,1)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and tid=(2,2) posting 2 (point to heap tid=(4,1) and tid=(4,4)) page lsn=0/4DAD7E0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 2 and posting 3 (point to heap tid=(4,4) and tid=(4,5)) page lsn=0/4DBD070.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 3 and posting 4 (point to heap tid=(4,5) and tid=(4,6)) page lsn=0/4DBD070.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 4 and tid=(2,3) (point to heap tid=(4,6) and tid=(5,1)) page lsn=0/4DBD070.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and tid=(4,3) posting 0 (point to heap tid=(5,6) and tid=(6,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(6,1) and tid=(6,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(6,2) and tid=(6,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(6,3) and tid=(6,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(6,4) and tid=(6,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and tid=(4,4) posting 0 (point to heap tid=(6,5) and tid=(6,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(6,6) and tid=(7,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(7,1) and tid=(7,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(7,2) and tid=(7,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(7,3) and tid=(7,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and tid=(4,5) posting 0 (point to heap tid=(7,4) and tid=(7,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(7,5) and tid=(7,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(7,6) and tid=(8,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(8,1) and tid=(8,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(8,2) and tid=(8,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) posting 0 (point to heap tid=(8,3) and tid=(8,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 0 and posting 1 (point to heap tid=(8,4) and tid=(8,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 1 and posting 2 (point to heap tid=(8,5) and tid=(8,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 2 and posting 3 (point to heap tid=(8,6) and tid=(9,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 3 and posting 4 (point to heap tid=(9,1) and tid=(9,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness may be violated for index "bttest_unique_idx": Index tid=(5,1) doesn't have visible heap tids and key is equal to the tid=(4,6) posting 4 (points to heap tid=(9,2)). Cross-page unique constraint violation can be missed. Vacuum the table and repeat the check.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,2) and tid=(5,3) (point to heap tid=(9,4) and tid=(9,5)) page lsn=0/4DC77A8.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,3) and tid=(5,4) (point to heap tid=(9,5) and tid=(9,6)) page lsn=0/4DC77A8.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,4) and tid=(5,5) (point to heap tid=(9,6) and tid=(10,1)) page lsn=0/4DC77A8.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,5) and tid=(5,6) (point to heap tid=(10,1) and tid=(10,2)) page lsn=0/4DC77A8.
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 1 (point to heap tid=(0,1) and tid=(0,3)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 1 and posting 2 (point to heap tid=(0,3) and tid=(0,4)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,4) and tid=(0,5)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and tid=(1,3) posting 0 (point to heap tid=(0,5) and tid=(0,6)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(0,6) and tid=(1,1)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,1) and tid=(1,2)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,2) and tid=(1,3)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,3) and tid=(1,4)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and tid=(1,4) posting 0 (point to heap tid=(1,4) and tid=(1,5)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(1,5) and tid=(1,6)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(1,6) and tid=(2,1)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,1) and tid=(2,2)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,2) and tid=(2,3)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and tid=(1,5) posting 0 (point to heap tid=(2,3) and tid=(2,4)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(2,4) and tid=(2,5)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(2,5) and tid=(2,6)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(2,6) and tid=(3,1)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,1) and tid=(3,2)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and tid=(1,6) posting 0 (point to heap tid=(3,2) and tid=(3,3)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(3,3) and tid=(3,4)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(3,4) and tid=(3,5)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(3,5) and tid=(3,6)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(3,6) and tid=(4,1)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and tid=(2,2) posting 0 (point to heap tid=(4,1) and tid=(4,4)) page lsn=0/4DC9D58.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 0 and posting 1 (point to heap tid=(4,4) and tid=(4,5)) page lsn=0/4DC9D98.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 1 and posting 2 (point to heap tid=(4,5) and tid=(4,6)) page lsn=0/4DC9D98.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 2 and tid=(2,3) (point to heap tid=(4,6) and tid=(5,1)) page lsn=0/4DC9D98.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and tid=(4,3) posting 0 (point to heap tid=(5,6) and tid=(6,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(6,1) and tid=(6,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(6,2) and tid=(6,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(6,3) and tid=(6,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(6,4) and tid=(6,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and tid=(4,4) posting 0 (point to heap tid=(6,5) and tid=(6,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(6,6) and tid=(7,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(7,1) and tid=(7,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(7,2) and tid=(7,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(7,3) and tid=(7,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and tid=(4,5) posting 0 (point to heap tid=(7,4) and tid=(7,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(7,5) and tid=(7,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(7,6) and tid=(8,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(8,1) and tid=(8,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(8,2) and tid=(8,3)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) posting 0 (point to heap tid=(8,3) and tid=(8,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 0 and posting 1 (point to heap tid=(8,4) and tid=(8,5)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 1 and posting 2 (point to heap tid=(8,5) and tid=(8,6)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 2 and posting 3 (point to heap tid=(8,6) and tid=(9,1)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 3 and posting 4 (point to heap tid=(9,1) and tid=(9,2)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 4 and tid=(5,1) (point to heap tid=(9,2) and tid=(9,4)) page lsn=0/4DC4CD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,1) and tid=(5,2) (point to heap tid=(9,4) and tid=(9,5)) page lsn=0/4DC9DD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,2) and tid=(5,3) (point to heap tid=(9,5) and tid=(9,6)) page lsn=0/4DC9DD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,3) and tid=(5,4) (point to heap tid=(9,6) and tid=(10,1)) page lsn=0/4DC9DD0.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,4) and tid=(5,5) (point to heap tid=(10,1) and tid=(10,2)) page lsn=0/4DC9DD0.
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..8df296431fb 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,11 +115,35 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+SELECT bt_index_check('bttest_unique_idx', true);
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index b8c7793d9e0..4519b1e0400 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -83,6 +83,13 @@ typedef struct BtreeCheckState
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking.
+ * Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -148,8 +155,21 @@ static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -187,6 +207,7 @@ static ItemId PageGetItemIdCareful(BtreeCheckState *state, BlockNumber block,
static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
IndexTuple itup, bool nonpivot);
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
+static bool errflag; /* Output ERROR at the end of amcheck */
/*
* bt_index_check(index regclass, heapallindexed boolean)
@@ -449,6 +470,15 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->indexinfo = BuildIndexInfo(state->rel);
+ /*
+ * We need a snapshot it to check uniqueness of the index
+ * For better performance, take it once per index check.
+ */
+ if (state->indexinfo->ii_Unique)
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ else
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -632,7 +662,16 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
+
+ if (errflag == true)
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index \"%s\" is corrupted. There are tuples violating UNIQUE constraint",
+ RelationGetRelationName(state->rel)),
+ errdetail_internal("Details are in the previous log messages under WARNING priority")));
}
/*
@@ -1006,6 +1045,152 @@ bt_recheck_sibling_links(BtreeCheckState *state,
btpo_prev_from_target)));
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ errflag = true;
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) "
+ "page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ (uint32) (state->targetlsn >> 32),
+ (uint32) state->targetlsn)));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+ /*
+ * Prevent double reporting unique violation between the posting
+ * list entries of a first tuple on the page after cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid (*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list.
+ * If TID is visible, save info about it for next comparisons in the loop in
+ * bt_page_check(). If also lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness may be violated for index \"%s\": "
+ "Index tid=(%u,%u) doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Cross-page unique constraint violation can be missed. "
+ "Vacuum the table and repeat the check.",
+ RelationGetRelationName(state->rel),
+ targetblock, offset,
+ *lVis_block, *lVis_offset, psprintf(" posting %u", *lVis_i),
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+}
+
/*
* Function performs the following checks on target page, or pages ancillary to
* target page:
@@ -1026,6 +1211,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1047,6 +1235,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber offset;
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for posting
+ * tuple. for non-posting tuple (-1)
+ */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1446,6 +1641,39 @@ bt_target_page_check(BtreeCheckState *state)
(uint32) state->targetlsn)));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking
+ * heap tuples visibility.
+ */
+ if (state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+ /* Invalidate scankey tid to make _bt_compare compare only keys
+ * in the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1463,12 +1691,14 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1503,6 +1733,43 @@ bt_target_page_check(BtreeCheckState *state)
(uint32) (state->targetlsn >> 32),
(uint32) state->targetlsn)));
}
+
+ /*
+ * If index has unique constraint check that not more than one found
+ * equal items is visible.
+ */
+ if (state->indexinfo->ii_Unique && rightkey && P_ISLEAF(topaque))
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ state->targetblock + 1);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, state->targetblock + 1,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, state->targetblock + 1, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1548,9 +1815,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1713,6 +1982,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
--
2.28.0
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-02-09 18:57 ` Pavel Borisov <[email protected]>
2 siblings, 0 replies; 45+ messages in thread
From: Pavel Borisov @ 2021-02-09 18:57 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>
To make tests stable I also removed lsn output under warning level. PFA v3
of a patch
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v3-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch (34.1K, ../../CALT9ZEEX033GE7G3bQCg7rw+Ky9VmtHU_J0c-PQwdLxTr882cQ@mail.gmail.com/3-v3-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch)
download | inline diff:
From 1dcf0206cabcaf967de28bc95b28f8587d36ffaa Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 8 Feb 2021 12:26:08 +0400
Subject: [PATCH v3] Make amcheck checking UNIQUE constraint for btree index.
On index with unique constraint ake check that only one table entry for the
equal keys (including all posting list entries) is visible. Report error if
not and show all index entries violating the constraint under warning level.
Authors: Anastasia Lubennikova <[email protected]>, Pavel Borisov <[email protected]>
---
contrib/amcheck/expected/check_btree.out | 138 ++++++++++++
contrib/amcheck/sql/check_btree.sql | 24 ++
contrib/amcheck/verify_nbtree.c | 275 ++++++++++++++++++++++-
3 files changed, 433 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 13848b7449b..fe4a958b1b0 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,11 +177,149 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+SELECT bt_index_check('bttest_unique_idx', true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 2 (point to heap tid=(0,1) and tid=(0,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,3) and tid=(0,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and posting 4 (point to heap tid=(0,4) and tid=(0,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 4 and tid=(1,3) posting 0 (point to heap tid=(0,5) and tid=(0,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(0,6) and tid=(1,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,1) and tid=(1,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,2) and tid=(1,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,3) and tid=(1,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and tid=(1,4) posting 0 (point to heap tid=(1,4) and tid=(1,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(1,5) and tid=(1,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(1,6) and tid=(2,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,1) and tid=(2,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,2) and tid=(2,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and tid=(1,5) posting 0 (point to heap tid=(2,3) and tid=(2,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(2,4) and tid=(2,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(2,5) and tid=(2,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(2,6) and tid=(3,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,1) and tid=(3,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and tid=(1,6) posting 0 (point to heap tid=(3,2) and tid=(3,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(3,3) and tid=(3,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(3,4) and tid=(3,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(3,5) and tid=(3,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(3,6) and tid=(4,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and tid=(2,2) posting 2 (point to heap tid=(4,1) and tid=(4,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 2 and posting 3 (point to heap tid=(4,4) and tid=(4,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 3 and posting 4 (point to heap tid=(4,5) and tid=(4,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 4 and tid=(2,3) (point to heap tid=(4,6) and tid=(5,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and tid=(4,3) posting 0 (point to heap tid=(5,6) and tid=(6,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(6,1) and tid=(6,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(6,2) and tid=(6,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(6,3) and tid=(6,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(6,4) and tid=(6,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and tid=(4,4) posting 0 (point to heap tid=(6,5) and tid=(6,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(6,6) and tid=(7,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(7,1) and tid=(7,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(7,2) and tid=(7,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(7,3) and tid=(7,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and tid=(4,5) posting 0 (point to heap tid=(7,4) and tid=(7,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(7,5) and tid=(7,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(7,6) and tid=(8,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(8,1) and tid=(8,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(8,2) and tid=(8,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) posting 0 (point to heap tid=(8,3) and tid=(8,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 0 and posting 1 (point to heap tid=(8,4) and tid=(8,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 1 and posting 2 (point to heap tid=(8,5) and tid=(8,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 2 and posting 3 (point to heap tid=(8,6) and tid=(9,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 3 and posting 4 (point to heap tid=(9,1) and tid=(9,2)).
+WARNING: index uniqueness may be violated for index "bttest_unique_idx": Index tid=(5,1) doesn't have visible heap tids and key is equal to the tid=(4,6) posting 4 (points to heap tid=(9,2)). Cross-page unique constraint violation can be missed. Vacuum the table and repeat the check.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,2) and tid=(5,3) (point to heap tid=(9,4) and tid=(9,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,3) and tid=(5,4) (point to heap tid=(9,5) and tid=(9,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,4) and tid=(5,5) (point to heap tid=(9,6) and tid=(10,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,5) and tid=(5,6) (point to heap tid=(10,1) and tid=(10,2)).
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 1 (point to heap tid=(0,1) and tid=(0,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 1 and posting 2 (point to heap tid=(0,3) and tid=(0,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,4) and tid=(0,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and tid=(1,3) posting 0 (point to heap tid=(0,5) and tid=(0,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(0,6) and tid=(1,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,1) and tid=(1,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,2) and tid=(1,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,3) and tid=(1,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and tid=(1,4) posting 0 (point to heap tid=(1,4) and tid=(1,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(1,5) and tid=(1,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(1,6) and tid=(2,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,1) and tid=(2,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,2) and tid=(2,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and tid=(1,5) posting 0 (point to heap tid=(2,3) and tid=(2,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(2,4) and tid=(2,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(2,5) and tid=(2,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(2,6) and tid=(3,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,1) and tid=(3,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and tid=(1,6) posting 0 (point to heap tid=(3,2) and tid=(3,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(3,3) and tid=(3,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(3,4) and tid=(3,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(3,5) and tid=(3,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(3,6) and tid=(4,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and tid=(2,2) posting 0 (point to heap tid=(4,1) and tid=(4,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 0 and posting 1 (point to heap tid=(4,4) and tid=(4,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 1 and posting 2 (point to heap tid=(4,5) and tid=(4,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 2 and tid=(2,3) (point to heap tid=(4,6) and tid=(5,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and tid=(4,3) posting 0 (point to heap tid=(5,6) and tid=(6,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(6,1) and tid=(6,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(6,2) and tid=(6,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(6,3) and tid=(6,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(6,4) and tid=(6,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and tid=(4,4) posting 0 (point to heap tid=(6,5) and tid=(6,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(6,6) and tid=(7,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(7,1) and tid=(7,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(7,2) and tid=(7,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(7,3) and tid=(7,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and tid=(4,5) posting 0 (point to heap tid=(7,4) and tid=(7,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(7,5) and tid=(7,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(7,6) and tid=(8,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(8,1) and tid=(8,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(8,2) and tid=(8,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) posting 0 (point to heap tid=(8,3) and tid=(8,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 0 and posting 1 (point to heap tid=(8,4) and tid=(8,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 1 and posting 2 (point to heap tid=(8,5) and tid=(8,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 2 and posting 3 (point to heap tid=(8,6) and tid=(9,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 3 and posting 4 (point to heap tid=(9,1) and tid=(9,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 4 and tid=(5,1) (point to heap tid=(9,2) and tid=(9,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,1) and tid=(5,2) (point to heap tid=(9,4) and tid=(9,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,2) and tid=(5,3) (point to heap tid=(9,5) and tid=(9,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,3) and tid=(5,4) (point to heap tid=(9,6) and tid=(10,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,4) and tid=(5,5) (point to heap tid=(10,1) and tid=(10,2)).
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..8df296431fb 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,11 +115,35 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+SELECT bt_index_check('bttest_unique_idx', true);
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index b8c7793d9e0..6136cf1a22b 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -83,6 +83,13 @@ typedef struct BtreeCheckState
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking.
+ * Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -148,8 +155,21 @@ static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -187,6 +207,7 @@ static ItemId PageGetItemIdCareful(BtreeCheckState *state, BlockNumber block,
static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
IndexTuple itup, bool nonpivot);
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
+static bool errflag; /* Output ERROR at the end of amcheck */
/*
* bt_index_check(index regclass, heapallindexed boolean)
@@ -449,6 +470,15 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->indexinfo = BuildIndexInfo(state->rel);
+ /*
+ * We need a snapshot it to check uniqueness of the index
+ * For better performance, take it once per index check.
+ */
+ if (state->indexinfo->ii_Unique)
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ else
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -632,7 +662,16 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
+
+ if (errflag == true)
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index \"%s\" is corrupted. There are tuples violating UNIQUE constraint",
+ RelationGetRelationName(state->rel)),
+ errdetail_internal("Details are in the previous log messages under WARNING priority")));
}
/*
@@ -1006,6 +1045,149 @@ bt_recheck_sibling_links(BtreeCheckState *state,
btpo_prev_from_target)));
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ errflag = true;
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s).",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid)));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+ /*
+ * Prevent double reporting unique violation between the posting
+ * list entries of a first tuple on the page after cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid (*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list.
+ * If TID is visible, save info about it for next comparisons in the loop in
+ * bt_page_check(). If also lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness may be violated for index \"%s\": "
+ "Index tid=(%u,%u) doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Cross-page unique constraint violation can be missed. "
+ "Vacuum the table and repeat the check.",
+ RelationGetRelationName(state->rel),
+ targetblock, offset,
+ *lVis_block, *lVis_offset, psprintf(" posting %u", *lVis_i),
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+}
+
/*
* Function performs the following checks on target page, or pages ancillary to
* target page:
@@ -1026,6 +1208,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1047,6 +1232,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber offset;
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for posting
+ * tuple. for non-posting tuple (-1)
+ */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1446,6 +1638,39 @@ bt_target_page_check(BtreeCheckState *state)
(uint32) state->targetlsn)));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking
+ * heap tuples visibility.
+ */
+ if (state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+ /* Invalidate scankey tid to make _bt_compare compare only keys
+ * in the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1463,12 +1688,14 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1503,6 +1730,43 @@ bt_target_page_check(BtreeCheckState *state)
(uint32) (state->targetlsn >> 32),
(uint32) state->targetlsn)));
}
+
+ /*
+ * If index has unique constraint check that not more than one found
+ * equal items is visible.
+ */
+ if (state->indexinfo->ii_Unique && rightkey && P_ISLEAF(topaque))
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ state->targetblock + 1);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, state->targetblock + 1,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, state->targetblock + 1, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1548,9 +1812,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1713,6 +1979,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
--
2.28.0
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-02-09 19:41 ` Zhihong Yu <[email protected]>
2 siblings, 0 replies; 45+ messages in thread
From: Zhihong Yu @ 2021-02-09 19:41 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Mark Dilger <[email protected]>; Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>
Hi,
Minor comment:
+ if (errflag == true)
+ ereport(ERROR,
I think 'if (errflag)' should suffice.
Cheers
On Tue, Feb 9, 2021 at 10:44 AM Pavel Borisov <[email protected]>
wrote:
> вт, 9 февр. 2021 г. в 01:46, Mark Dilger <[email protected]>:
>
>>
>>
>> > On Feb 8, 2021, at 2:46 AM, Pavel Borisov <[email protected]>
>> wrote:
>> >
>> > 0002 - is a temporary hack for testing. It will allow inserting
>> duplicates in a table even if an index with the exact name "idx" has a
>> unique constraint (generally it is prohibited to insert). Then a new
>> amcheck will tell us about these duplicates. It's pity but testing can not
>> be done automatically, as it needs a core recompile. For testing I'd
>> recommend a protocol similar to the following:
>> >
>> > - Apply patch 0002
>> > - Set autovaccum = off in postgresql.conf
>>
>> Thanks Pavel and Anastasia for working on this!
>>
>> Updating pg_catalog directly is ugly, but the following seems a simpler
>> way to set up a regression test than having to recompile. What do you
>> think?
>>
>> Very nice idea, thanks!
> I've made a regression test based on it. PFA v.2 of a patch. Now it
> doesn't need anything external for testing.
>
> --
> Best regards,
> Pavel Borisov
>
> Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
>
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-03-01 19:21 ` Mark Dilger <[email protected]>
2021-03-01 20:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-03 02:54 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Geoghegan <[email protected]>
2 siblings, 3 replies; 45+ messages in thread
From: Mark Dilger @ 2021-03-01 19:21 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
> On Feb 9, 2021, at 10:43 AM, Pavel Borisov <[email protected]> wrote:
>
> вт, 9 февр. 2021 г. в 01:46, Mark Dilger <[email protected]>:
>
>
> > On Feb 8, 2021, at 2:46 AM, Pavel Borisov <[email protected]> wrote:
> >
> > 0002 - is a temporary hack for testing. It will allow inserting duplicates in a table even if an index with the exact name "idx" has a unique constraint (generally it is prohibited to insert). Then a new amcheck will tell us about these duplicates. It's pity but testing can not be done automatically, as it needs a core recompile. For testing I'd recommend a protocol similar to the following:
> >
> > - Apply patch 0002
> > - Set autovaccum = off in postgresql.conf
>
> Thanks Pavel and Anastasia for working on this!
>
> Updating pg_catalog directly is ugly, but the following seems a simpler way to set up a regression test than having to recompile. What do you think?
>
> Very nice idea, thanks!
> I've made a regression test based on it. PFA v.2 of a patch. Now it doesn't need anything external for testing.
If bt_index_check() and bt_index_parent_check() are to have this functionality, shouldn't there be an option controlling it much as the option (heapallindexed boolean) controls checking whether all entries in the heap are indexed in the btree? It seems inconsistent to have an option to avoid checking the heap for that, but not for this. Alternately, this might even be better coded as its own function, named something like bt_unique_index_check() perhaps. I hope Peter might advise?
The regression test you provided is not portable. I am getting lots of errors due to differing output of the form "page lsn=0/4DAD7E0". You might turn this into a TAP test and use a regular expression to check the output.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-03-01 20:05 ` Pavel Borisov <[email protected]>
2021-03-01 20:20 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2021-03-01 20:05 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
>
> The regression test you provided is not portable. I am getting lots of
> errors due to differing output of the form "page lsn=0/4DAD7E0". You might
> turn this into a TAP test and use a regular expression to check the output.
>
May I ask you to ensure you used v3 of a patch to check? I've made tests
portable in v3, probably, you've checked not the last version.
Thanks for your attention to the patch
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-03-01 20:20 ` Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Tom Lane <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Mark Dilger @ 2021-03-01 20:20 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
> On Mar 1, 2021, at 12:05 PM, Pavel Borisov <[email protected]> wrote:
>
> The regression test you provided is not portable. I am getting lots of errors due to differing output of the form "page lsn=0/4DAD7E0". You might turn this into a TAP test and use a regular expression to check the output.
> May I ask you to ensure you used v3 of a patch to check? I've made tests portable in v3, probably, you've checked not the last version.
Yes, my review was of v2. Updating to v3, I see that the test passes on my laptop. It still looks brittle to have all the tid values in the test output, but it does pass.
> Thanks for your attention to the patch
Thanks for the patch!
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 20:20 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-03-01 20:23 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Tom Lane @ 2021-03-01 20:23 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
Mark Dilger <[email protected]> writes:
> Yes, my review was of v2. Updating to v3, I see that the test passes on my laptop. It still looks brittle to have all the tid values in the test output, but it does pass.
Hm, anyone tried it on 32-bit hardware?
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-03-01 20:23 ` Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2021-03-01 20:23 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
>
> If bt_index_check() and bt_index_parent_check() are to have this
> functionality, shouldn't there be an option controlling it much as the
> option (heapallindexed boolean) controls checking whether all entries in
> the heap are indexed in the btree? It seems inconsistent to have an option
> to avoid checking the heap for that, but not for this. Alternately, this
> might even be better coded as its own function, named something like
> bt_unique_index_check() perhaps. I hope Peter might advise?
>
As for heap checking, my reasoning was that we can not check whether a
unique constraint violated by the index, without checking heap tuple
visibility. I.e. we can have many identical index entries without
uniqueness violated if only one of them corresponds to a visible heap
tuple. So heap checking included in my patch is _necessary_ for unique
constraint checking, it should not have an option to be disabled,
otherwise, the only answer we can get is that unique constraint MAY be
violated and may not be, which is quite useless. If you meant something
different, please elaborate.
I can try to rewrite unique constraint checking to be done after all others
check but I suppose it's the performance considerations are that made
previous amcheck routines to do many checks simultaneously. I tried to
stick to this practice. It's also not so elegant to duplicate much code to
make uniqueness checks independently and the resulting patch will be much
bigger and harder to review.
Anyway, your and Peter's further considerations are always welcome.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-03-01 21:05 ` Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Mark Dilger @ 2021-03-01 21:05 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
> On Mar 1, 2021, at 12:23 PM, Pavel Borisov <[email protected]> wrote:
>
> If bt_index_check() and bt_index_parent_check() are to have this functionality, shouldn't there be an option controlling it much as the option (heapallindexed boolean) controls checking whether all entries in the heap are indexed in the btree? It seems inconsistent to have an option to avoid checking the heap for that, but not for this. Alternately, this might even be better coded as its own function, named something like bt_unique_index_check() perhaps. I hope Peter might advise?
>
> As for heap checking, my reasoning was that we can not check whether a unique constraint violated by the index, without checking heap tuple visibility. I.e. we can have many identical index entries without uniqueness violated if only one of them corresponds to a visible heap tuple. So heap checking included in my patch is _necessary_ for unique constraint checking, it should not have an option to be disabled, otherwise, the only answer we can get is that unique constraint MAY be violated and may not be, which is quite useless. If you meant something different, please elaborate.
I completely agree that checking uniqueness requires looking at the heap, but I don't agree that every caller of bt_index_check on an index wants that particular check to be performed. There are multiple ways in which an index might be corrupt, and Peter wrote the code to only check some of them by default, with options to expand the checks to other things. This is why heapallindexed is optional. If you don't want to pay the price of checking all entries in the heap against the btree, you don't have to.
I'm not against running uniqueness checks on unique indexes. It seems fairly normal that a user would want that. Perhaps the option should default to 'true' if unspecified? But having no option at all seems to run contrary to how the other functionality is structured.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-03-02 14:08 ` Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2021-03-02 14:08 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
>
> I completely agree that checking uniqueness requires looking at the heap,
> but I don't agree that every caller of bt_index_check on an index wants
> that particular check to be performed. There are multiple ways in which an
> index might be corrupt, and Peter wrote the code to only check some of them
> by default, with options to expand the checks to other things. This is why
> heapallindexed is optional. If you don't want to pay the price of checking
> all entries in the heap against the btree, you don't have to.
>
I've got the idea and revised the patch accordingly. Thanks!
Pfa v4 of a patch. I've added an optional argument to allow uniqueness
checks for the unique indexes.
Also, I added a test variant to make them work on 32-bit systems.
Unfortunately, converting the regression test to TAP would be a pain for
me. Hope it can be used now as a 2-variant regression test for 32 and 64
bit systems.
Thank you for your consideration!
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v4-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch (65.4K, ../../CALT9ZEFMb96zVafrt7sUPrWYMDQVngaPcXuVqXCfawA2cfWwxw@mail.gmail.com/3-v4-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch)
download | inline diff:
From 34ed10d11a479cbab8afed224f1718de099d0f35 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 8 Feb 2021 12:26:08 +0400
Subject: [PATCH v4] Make amcheck checking UNIQUE constraint for btree index.
On index with unique constraint ake check that only one table entry for the
equal keys (including all posting list entries) is visible. Report error if
not and show all index entries violating the constraint under warning level.
Authors: Anastasia Lubennikova <[email protected]>, Pavel Borisov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 28 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 146 +++++++++
contrib/amcheck/expected/check_btree_1.out | 333 +++++++++++++++++++++
contrib/amcheck/sql/check_btree.sql | 27 ++
contrib/amcheck/verify_nbtree.c | 311 ++++++++++++++++++-
7 files changed, 832 insertions(+), 17 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/expected/check_btree_1.out
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..f76e6941ffe
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,28 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded version of the 1.3 function signature
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 5a3f1ef737c..b191a4df862 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,11 +177,157 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+-- Check unique index with no uniqueness check. Should not complain.
+SELECT bt_index_check('bttest_unique_idx', true);
+ bt_index_check
+----------------
+
+(1 row)
+
+-- Check unique indes with uniquensee check. Should detect constraint violation cases.
+SELECT bt_index_check('bttest_unique_idx', true, true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 2 (point to heap tid=(0,1) and tid=(0,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,3) and tid=(0,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and posting 4 (point to heap tid=(0,4) and tid=(0,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 4 and tid=(1,3) posting 0 (point to heap tid=(0,5) and tid=(0,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(0,6) and tid=(1,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,1) and tid=(1,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,2) and tid=(1,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,3) and tid=(1,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and tid=(1,4) posting 0 (point to heap tid=(1,4) and tid=(1,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(1,5) and tid=(1,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(1,6) and tid=(2,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,1) and tid=(2,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,2) and tid=(2,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and tid=(1,5) posting 0 (point to heap tid=(2,3) and tid=(2,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(2,4) and tid=(2,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(2,5) and tid=(2,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(2,6) and tid=(3,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,1) and tid=(3,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and tid=(1,6) posting 0 (point to heap tid=(3,2) and tid=(3,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(3,3) and tid=(3,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(3,4) and tid=(3,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(3,5) and tid=(3,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(3,6) and tid=(4,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and tid=(2,2) posting 2 (point to heap tid=(4,1) and tid=(4,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 2 and posting 3 (point to heap tid=(4,4) and tid=(4,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 3 and posting 4 (point to heap tid=(4,5) and tid=(4,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 4 and tid=(2,3) (point to heap tid=(4,6) and tid=(5,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and tid=(4,3) posting 0 (point to heap tid=(5,6) and tid=(6,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(6,1) and tid=(6,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(6,2) and tid=(6,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(6,3) and tid=(6,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(6,4) and tid=(6,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and tid=(4,4) posting 0 (point to heap tid=(6,5) and tid=(6,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(6,6) and tid=(7,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(7,1) and tid=(7,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(7,2) and tid=(7,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(7,3) and tid=(7,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and tid=(4,5) posting 0 (point to heap tid=(7,4) and tid=(7,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(7,5) and tid=(7,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(7,6) and tid=(8,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(8,1) and tid=(8,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(8,2) and tid=(8,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) posting 0 (point to heap tid=(8,3) and tid=(8,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 0 and posting 1 (point to heap tid=(8,4) and tid=(8,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 1 and posting 2 (point to heap tid=(8,5) and tid=(8,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 2 and posting 3 (point to heap tid=(8,6) and tid=(9,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 3 and posting 4 (point to heap tid=(9,1) and tid=(9,2)).
+WARNING: index uniqueness may be violated for index "bttest_unique_idx": Index tid=(5,1) doesn't have visible heap tids and key is equal to the tid=(4,6) posting 4 (points to heap tid=(9,2)). Cross-page unique constraint violation can be missed. Vacuum the table and repeat the check.
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,2) and tid=(5,3) (point to heap tid=(9,4) and tid=(9,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,3) and tid=(5,4) (point to heap tid=(9,5) and tid=(9,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,4) and tid=(5,5) (point to heap tid=(9,6) and tid=(10,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,5) and tid=(5,6) (point to heap tid=(10,1) and tid=(10,2)).
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true, true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 1 (point to heap tid=(0,1) and tid=(0,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 1 and posting 2 (point to heap tid=(0,3) and tid=(0,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,4) and tid=(0,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and tid=(1,3) posting 0 (point to heap tid=(0,5) and tid=(0,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(0,6) and tid=(1,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,1) and tid=(1,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,2) and tid=(1,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,3) and tid=(1,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and tid=(1,4) posting 0 (point to heap tid=(1,4) and tid=(1,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(1,5) and tid=(1,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(1,6) and tid=(2,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,1) and tid=(2,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,2) and tid=(2,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and tid=(1,5) posting 0 (point to heap tid=(2,3) and tid=(2,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(2,4) and tid=(2,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(2,5) and tid=(2,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(2,6) and tid=(3,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,1) and tid=(3,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and tid=(1,6) posting 0 (point to heap tid=(3,2) and tid=(3,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(3,3) and tid=(3,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(3,4) and tid=(3,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(3,5) and tid=(3,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(3,6) and tid=(4,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and tid=(2,2) posting 0 (point to heap tid=(4,1) and tid=(4,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 0 and posting 1 (point to heap tid=(4,4) and tid=(4,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 1 and posting 2 (point to heap tid=(4,5) and tid=(4,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(2,2) posting 2 and tid=(2,3) (point to heap tid=(4,6) and tid=(5,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and tid=(4,3) posting 0 (point to heap tid=(5,6) and tid=(6,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(6,1) and tid=(6,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(6,2) and tid=(6,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(6,3) and tid=(6,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(6,4) and tid=(6,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and tid=(4,4) posting 0 (point to heap tid=(6,5) and tid=(6,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(6,6) and tid=(7,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(7,1) and tid=(7,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(7,2) and tid=(7,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(7,3) and tid=(7,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and tid=(4,5) posting 0 (point to heap tid=(7,4) and tid=(7,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(7,5) and tid=(7,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(7,6) and tid=(8,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(8,1) and tid=(8,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(8,2) and tid=(8,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) posting 0 (point to heap tid=(8,3) and tid=(8,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 0 and posting 1 (point to heap tid=(8,4) and tid=(8,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 1 and posting 2 (point to heap tid=(8,5) and tid=(8,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 2 and posting 3 (point to heap tid=(8,6) and tid=(9,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 3 and posting 4 (point to heap tid=(9,1) and tid=(9,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,6) posting 4 and tid=(5,1) (point to heap tid=(9,2) and tid=(9,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,1) and tid=(5,2) (point to heap tid=(9,4) and tid=(9,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,2) and tid=(5,3) (point to heap tid=(9,5) and tid=(9,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,3) and tid=(5,4) (point to heap tid=(9,6) and tid=(10,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(5,4) and tid=(5,5) (point to heap tid=(10,1) and tid=(10,2)).
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/expected/check_btree_1.out b/contrib/amcheck/expected/check_btree_1.out
new file mode 100644
index 00000000000..6aa66e42118
--- /dev/null
+++ b/contrib/amcheck/expected/check_btree_1.out
@@ -0,0 +1,333 @@
+CREATE TABLE bttest_a(id int8);
+CREATE TABLE bttest_b(id int8);
+CREATE TABLE bttest_multi(id int8, data int8);
+CREATE TABLE delete_test_table (a bigint, b bigint, c bigint, d bigint);
+-- Stabalize tests
+ALTER TABLE bttest_a SET (autovacuum_enabled = false);
+ALTER TABLE bttest_b SET (autovacuum_enabled = false);
+ALTER TABLE bttest_multi SET (autovacuum_enabled = false);
+ALTER TABLE delete_test_table SET (autovacuum_enabled = false);
+INSERT INTO bttest_a SELECT * FROM generate_series(1, 100000);
+INSERT INTO bttest_b SELECT * FROM generate_series(100000, 1, -1);
+INSERT INTO bttest_multi SELECT i, i%2 FROM generate_series(1, 100000) as i;
+CREATE INDEX bttest_a_idx ON bttest_a USING btree (id) WITH (deduplicate_items = ON);
+CREATE INDEX bttest_b_idx ON bttest_b USING btree (id);
+CREATE UNIQUE INDEX bttest_multi_idx ON bttest_multi
+USING btree (id) INCLUDE (data);
+CREATE ROLE regress_bttest_role;
+-- verify permissions are checked (error due to function not callable)
+SET ROLE regress_bttest_role;
+SELECT bt_index_check('bttest_a_idx'::regclass);
+ERROR: permission denied for function bt_index_check
+SELECT bt_index_parent_check('bttest_a_idx'::regclass);
+ERROR: permission denied for function bt_index_parent_check
+RESET ROLE;
+-- we, intentionally, don't check relation permissions - it's useful
+-- to run this cluster-wide with a restricted account, and as tested
+-- above explicit permission has to be granted for that.
+GRANT EXECUTE ON FUNCTION bt_index_check(regclass) TO regress_bttest_role;
+GRANT EXECUTE ON FUNCTION bt_index_parent_check(regclass) TO regress_bttest_role;
+GRANT EXECUTE ON FUNCTION bt_index_check(regclass, boolean) TO regress_bttest_role;
+GRANT EXECUTE ON FUNCTION bt_index_parent_check(regclass, boolean) TO regress_bttest_role;
+SET ROLE regress_bttest_role;
+SELECT bt_index_check('bttest_a_idx');
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx');
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+RESET ROLE;
+-- verify plain tables are rejected (error)
+SELECT bt_index_check('bttest_a');
+ERROR: "bttest_a" is not an index
+SELECT bt_index_parent_check('bttest_a');
+ERROR: "bttest_a" is not an index
+-- verify non-existing indexes are rejected (error)
+SELECT bt_index_check(17);
+ERROR: could not open relation with OID 17
+SELECT bt_index_parent_check(17);
+ERROR: could not open relation with OID 17
+-- verify wrong index types are rejected (error)
+BEGIN;
+CREATE INDEX bttest_a_brin_idx ON bttest_a USING brin(id);
+SELECT bt_index_parent_check('bttest_a_brin_idx');
+ERROR: only B-Tree indexes are supported as targets for verification
+DETAIL: Relation "bttest_a_brin_idx" is not a B-Tree index.
+ROLLBACK;
+-- normal check outside of xact
+SELECT bt_index_check('bttest_a_idx');
+ bt_index_check
+----------------
+
+(1 row)
+
+-- more expansive tests
+SELECT bt_index_check('bttest_a_idx', true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+BEGIN;
+SELECT bt_index_check('bttest_a_idx');
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx');
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- make sure we don't have any leftover locks
+SELECT * FROM pg_locks
+WHERE relation = ANY(ARRAY['bttest_a', 'bttest_a_idx', 'bttest_b', 'bttest_b_idx']::regclass[])
+ AND pid = pg_backend_pid();
+ locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath | waitstart
+----------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-----+------+---------+----------+-----------
+(0 rows)
+
+COMMIT;
+-- Deduplication
+TRUNCATE bttest_a;
+INSERT INTO bttest_a SELECT 42 FROM generate_series(1, 2000);
+SELECT bt_index_check('bttest_a_idx', true);
+ bt_index_check
+----------------
+
+(1 row)
+
+-- normal check outside of xact for index with included columns
+SELECT bt_index_check('bttest_multi_idx');
+ bt_index_check
+----------------
+
+(1 row)
+
+-- more expansive tests for index with included columns
+SELECT bt_index_parent_check('bttest_multi_idx', true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- repeat expansive tests for index built using insertions
+TRUNCATE bttest_multi;
+INSERT INTO bttest_multi SELECT i, i%2 FROM generate_series(1, 100000) as i;
+SELECT bt_index_parent_check('bttest_multi_idx', true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+--
+-- Test for multilevel page deletion/downlink present checks, and rootdescend
+-- checks
+--
+INSERT INTO delete_test_table SELECT i, 1, 2, 3 FROM generate_series(1,80000) i;
+ALTER TABLE delete_test_table ADD PRIMARY KEY (a,b,c,d);
+-- Delete most entries, and vacuum, deleting internal pages and creating "fast
+-- root"
+DELETE FROM delete_test_table WHERE a < 79990;
+VACUUM delete_test_table;
+SELECT bt_index_parent_check('delete_test_table_pkey', true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+--
+-- BUG #15597: must not assume consistent input toasting state when forming
+-- tuple. Bloom filter must fingerprint normalized index tuple representation.
+--
+CREATE TABLE toast_bug(buggy text);
+ALTER TABLE toast_bug ALTER COLUMN buggy SET STORAGE extended;
+CREATE INDEX toasty ON toast_bug(buggy);
+-- pg_attribute entry for toasty.buggy (the index) will have plain storage:
+UPDATE pg_attribute SET attstorage = 'p'
+WHERE attrelid = 'toasty'::regclass AND attname = 'buggy';
+-- Whereas pg_attribute entry for toast_bug.buggy (the table) still has extended storage:
+SELECT attstorage FROM pg_attribute
+WHERE attrelid = 'toast_bug'::regclass AND attname = 'buggy';
+ attstorage
+------------
+ x
+(1 row)
+
+-- Insert compressible heap tuple (comfortably exceeds TOAST_TUPLE_THRESHOLD):
+INSERT INTO toast_bug SELECT repeat('a', 2200);
+-- Should not get false positive report of corruption:
+SELECT bt_index_check('toasty', true);
+ bt_index_check
+----------------
+
+(1 row)
+
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+-- Check unique index with no uniqueness check. Should not complain.
+SELECT bt_index_check('bttest_unique_idx', true);
+ bt_index_check
+----------------
+
+(1 row)
+
+-- Check unique indes with uniquensee check. Should detect constraint violation cases.
+SELECT bt_index_check('bttest_unique_idx', true, true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 2 (point to heap tid=(0,1) and tid=(0,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,3) and tid=(0,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and posting 4 (point to heap tid=(0,4) and tid=(0,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 4 and posting 5 (point to heap tid=(0,5) and tid=(0,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 5 and tid=(1,3) posting 0 (point to heap tid=(0,6) and tid=(1,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(1,1) and tid=(1,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,2) and tid=(1,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,3) and tid=(1,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,4) and tid=(1,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and posting 5 (point to heap tid=(1,5) and tid=(1,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 5 and tid=(1,4) posting 0 (point to heap tid=(1,6) and tid=(2,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(2,1) and tid=(2,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(2,2) and tid=(2,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,3) and tid=(2,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,4) and tid=(2,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and posting 5 (point to heap tid=(2,5) and tid=(2,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 5 and tid=(1,5) posting 0 (point to heap tid=(2,6) and tid=(3,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(3,1) and tid=(3,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(3,2) and tid=(3,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(3,3) and tid=(3,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,4) and tid=(3,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and posting 5 (point to heap tid=(3,5) and tid=(3,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 5 and tid=(1,6) posting 0 (point to heap tid=(3,6) and tid=(4,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 3 (point to heap tid=(4,1) and tid=(4,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and posting 4 (point to heap tid=(4,4) and tid=(4,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 4 and posting 5 (point to heap tid=(4,5) and tid=(4,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 5 and tid=(2,2) (point to heap tid=(4,6) and tid=(5,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 4 and posting 5 (point to heap tid=(5,6) and tid=(6,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 5 and tid=(4,2) posting 0 (point to heap tid=(6,1) and tid=(6,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(6,2) and tid=(6,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(6,3) and tid=(6,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(6,4) and tid=(6,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(6,5) and tid=(6,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and posting 5 (point to heap tid=(6,6) and tid=(7,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 5 and tid=(4,3) posting 0 (point to heap tid=(7,1) and tid=(7,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(7,2) and tid=(7,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(7,3) and tid=(7,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(7,4) and tid=(7,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(7,5) and tid=(7,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and posting 5 (point to heap tid=(7,6) and tid=(8,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 5 and tid=(4,4) posting 0 (point to heap tid=(8,1) and tid=(8,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(8,2) and tid=(8,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(8,3) and tid=(8,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(8,4) and tid=(8,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(8,5) and tid=(8,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and posting 5 (point to heap tid=(8,6) and tid=(9,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 5 and tid=(4,5) posting 0 (point to heap tid=(9,1) and tid=(9,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 2 (point to heap tid=(9,2) and tid=(9,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(9,4) and tid=(9,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(9,5) and tid=(9,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and posting 5 (point to heap tid=(9,6) and tid=(10,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 5 and tid=(4,6) (point to heap tid=(10,1) and tid=(10,2)).
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true, true);
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 0 and posting 1 (point to heap tid=(0,1) and tid=(0,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 1 and posting 2 (point to heap tid=(0,3) and tid=(0,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 2 and posting 3 (point to heap tid=(0,4) and tid=(0,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 3 and posting 4 (point to heap tid=(0,5) and tid=(0,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,2) posting 4 and tid=(1,3) posting 0 (point to heap tid=(0,6) and tid=(1,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 0 and posting 1 (point to heap tid=(1,1) and tid=(1,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 1 and posting 2 (point to heap tid=(1,2) and tid=(1,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 2 and posting 3 (point to heap tid=(1,3) and tid=(1,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 3 and posting 4 (point to heap tid=(1,4) and tid=(1,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 4 and posting 5 (point to heap tid=(1,5) and tid=(1,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,3) posting 5 and tid=(1,4) posting 0 (point to heap tid=(1,6) and tid=(2,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 0 and posting 1 (point to heap tid=(2,1) and tid=(2,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 1 and posting 2 (point to heap tid=(2,2) and tid=(2,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 2 and posting 3 (point to heap tid=(2,3) and tid=(2,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 3 and posting 4 (point to heap tid=(2,4) and tid=(2,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 4 and posting 5 (point to heap tid=(2,5) and tid=(2,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,4) posting 5 and tid=(1,5) posting 0 (point to heap tid=(2,6) and tid=(3,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 0 and posting 1 (point to heap tid=(3,1) and tid=(3,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 1 and posting 2 (point to heap tid=(3,2) and tid=(3,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 2 and posting 3 (point to heap tid=(3,3) and tid=(3,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 3 and posting 4 (point to heap tid=(3,4) and tid=(3,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 4 and posting 5 (point to heap tid=(3,5) and tid=(3,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,5) posting 5 and tid=(1,6) posting 0 (point to heap tid=(3,6) and tid=(4,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 0 and posting 1 (point to heap tid=(4,1) and tid=(4,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 1 and posting 2 (point to heap tid=(4,4) and tid=(4,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 2 and posting 3 (point to heap tid=(4,5) and tid=(4,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(1,6) posting 3 and tid=(2,2) (point to heap tid=(4,6) and tid=(5,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 0 and posting 1 (point to heap tid=(5,2) and tid=(5,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 1 and posting 2 (point to heap tid=(5,3) and tid=(5,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 2 and posting 3 (point to heap tid=(5,4) and tid=(5,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 3 and posting 4 (point to heap tid=(5,5) and tid=(5,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 4 and posting 5 (point to heap tid=(5,6) and tid=(6,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,1) posting 5 and tid=(4,2) posting 0 (point to heap tid=(6,1) and tid=(6,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 0 and posting 1 (point to heap tid=(6,2) and tid=(6,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 1 and posting 2 (point to heap tid=(6,3) and tid=(6,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 2 and posting 3 (point to heap tid=(6,4) and tid=(6,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 3 and posting 4 (point to heap tid=(6,5) and tid=(6,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 4 and posting 5 (point to heap tid=(6,6) and tid=(7,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,2) posting 5 and tid=(4,3) posting 0 (point to heap tid=(7,1) and tid=(7,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 0 and posting 1 (point to heap tid=(7,2) and tid=(7,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 1 and posting 2 (point to heap tid=(7,3) and tid=(7,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 2 and posting 3 (point to heap tid=(7,4) and tid=(7,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 3 and posting 4 (point to heap tid=(7,5) and tid=(7,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 4 and posting 5 (point to heap tid=(7,6) and tid=(8,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,3) posting 5 and tid=(4,4) posting 0 (point to heap tid=(8,1) and tid=(8,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 0 and posting 1 (point to heap tid=(8,2) and tid=(8,3)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 1 and posting 2 (point to heap tid=(8,3) and tid=(8,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 2 and posting 3 (point to heap tid=(8,4) and tid=(8,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 3 and posting 4 (point to heap tid=(8,5) and tid=(8,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 4 and posting 5 (point to heap tid=(8,6) and tid=(9,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,4) posting 5 and tid=(4,5) posting 0 (point to heap tid=(9,1) and tid=(9,2)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 0 and posting 1 (point to heap tid=(9,2) and tid=(9,4)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 1 and posting 2 (point to heap tid=(9,4) and tid=(9,5)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 2 and posting 3 (point to heap tid=(9,5) and tid=(9,6)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 3 and posting 4 (point to heap tid=(9,6) and tid=(10,1)).
+WARNING: index uniqueness is violated for index "bttest_unique_idx": Index tid=(4,5) posting 4 and tid=(4,6) (point to heap tid=(10,1) and tid=(10,2)).
+ERROR: index "bttest_unique_idx" is corrupted. There are tuples violating UNIQUE constraint
+DETAIL: Details are in the previous log messages under WARNING priority
+-- cleanup
+DROP TABLE bttest_a;
+DROP TABLE bttest_b;
+DROP TABLE bttest_multi;
+DROP TABLE delete_test_table;
+DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
+DROP OWNED BY regress_bttest_role; -- permissions
+DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..dbe8b48afc7 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,11 +115,38 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+CREATE TABLE bttest_unique(a varchar(50), b varchar(1500), c bytea, d varchar(50));
+CREATE UNIQUE INDEX bttest_unique_idx ON bttest_unique(a,b);
+UPDATE pg_catalog.pg_index SET indisunique = false
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+INSERT INTO bttest_unique
+ SELECT i::text::varchar,
+ array_to_string(array(
+ SELECT substr('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', ((random()*(36-1)+1)::integer), 1)
+ FROM generate_series(1,1300)),'')::varchar,
+ i::text::bytea, i::text::varchar
+ FROM generate_series(0,1) AS i, generate_series(0,30) AS x;
+UPDATE pg_catalog.pg_index SET indisunique = true
+WHERE indrelid = (SELECT oid FROM pg_catalog.pg_class WHERE relname = 'bttest_unique');
+
+DELETE FROM bttest_unique WHERE ctid::text='(0,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,2)';
+DELETE FROM bttest_unique WHERE ctid::text='(4,3)';
+DELETE FROM bttest_unique WHERE ctid::text='(9,3)';
+-- Check unique index with no uniqueness check. Should not complain.
+SELECT bt_index_check('bttest_unique_idx', true);
+-- Check unique indes with uniquensee check. Should detect constraint violation cases.
+SELECT bt_index_check('bttest_unique_idx', true, true);
+VACUUM bttest_unique;
+SELECT bt_index_check('bttest_unique_idx', true, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index c4ca6339182..68644fa3285 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -78,11 +78,20 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking.
+ * Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -137,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -187,9 +210,10 @@ static ItemId PageGetItemIdCareful(BtreeCheckState *state, BlockNumber block,
static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
IndexTuple itup, bool nonpivot);
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
+static bool errflag; /* Output ERROR at the end of amcheck */
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -202,17 +226,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -226,13 +253,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -242,7 +272,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -323,7 +353,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/*
@@ -417,7 +447,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -449,6 +480,18 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
+ /*
+ * We need a snapshot it to check uniqueness of the index
+ * For better performance, take it once per index check.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
if (state->heapallindexed)
{
@@ -632,7 +675,16 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
+
+ if (errflag)
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index \"%s\" is corrupted. There are tuples violating UNIQUE constraint",
+ RelationGetRelationName(state->rel)),
+ errdetail_internal("Details are in the previous log messages under WARNING priority")));
}
/*
@@ -1006,6 +1058,149 @@ bt_recheck_sibling_links(BtreeCheckState *state,
btpo_prev_from_target)));
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ errflag = true;
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s).",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid)));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+ /*
+ * Prevent double reporting unique violation between the posting
+ * list entries of a first tuple on the page after cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid (*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list.
+ * If TID is visible, save info about it for next comparisons in the loop in
+ * bt_page_check(). If also lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid (*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ ereport(WARNING,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness may be violated for index \"%s\": "
+ "Index tid=(%u,%u) doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Cross-page unique constraint violation can be missed. "
+ "Vacuum the table and repeat the check.",
+ RelationGetRelationName(state->rel),
+ targetblock, offset,
+ *lVis_block, *lVis_offset, psprintf(" posting %u", *lVis_i),
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+}
+
/*
* Function performs the following checks on target page, or pages ancillary to
* target page:
@@ -1026,6 +1221,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1047,6 +1245,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber offset;
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for posting
+ * tuple. for non-posting tuple (-1)
+ */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1438,6 +1643,39 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking
+ * heap tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+ /* Invalidate scankey tid to make _bt_compare compare only keys
+ * in the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1455,12 +1693,14 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1494,6 +1734,44 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one found
+ * equal items is visible.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque))
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ state->targetblock + 1);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, state->targetblock + 1,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, state->targetblock + 1, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1539,9 +1817,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1704,6 +1984,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
--
2.28.0
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-03-15 15:11 ` Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Mark Dilger @ 2021-03-15 15:11 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>
> On Mar 2, 2021, at 6:08 AM, Pavel Borisov <[email protected]> wrote:
>
> I completely agree that checking uniqueness requires looking at the heap, but I don't agree that every caller of bt_index_check on an index wants that particular check to be performed. There are multiple ways in which an index might be corrupt, and Peter wrote the code to only check some of them by default, with options to expand the checks to other things. This is why heapallindexed is optional. If you don't want to pay the price of checking all entries in the heap against the btree, you don't have to.
>
> I've got the idea and revised the patch accordingly. Thanks!
> Pfa v4 of a patch. I've added an optional argument to allow uniqueness checks for the unique indexes.
> Also, I added a test variant to make them work on 32-bit systems. Unfortunately, converting the regression test to TAP would be a pain for me. Hope it can be used now as a 2-variant regression test for 32 and 64 bit systems.
>
> Thank you for your consideration!
>
> --
> Best regards,
> Pavel Borisov
>
> Postgres Professional: http://postgrespro.com
> <v4-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch>
Looking over v4, here are my review comments...
I created the file contrib/amcheck/amcheck--1.2--1.3.sql during the v14 development cycle, so that is not released yet. If your patch goes out in v14, does it need to create an amcheck--1.3--1.4.sql, or could you fit your changes into the 1.2--1.3.sql file? (Does the project have a convention governing this?) This is purely a question. I'm not advising you to change anything here.
You need to update doc/src/sgml/amcheck.sgml to reflect the changes you made to the bt_index_check and bt_index_parent_check functions.
You need to update the recently committed src/bin/pg_amcheck project to include --checkunique as an option. This client application already has flags for heapallindexed and rootdescend. I can help with that if it isn't obvious what needs to be done. Note that pg_amcheck/t contains TAP tests that exercise the options, so you'll need to extend code coverage to include this new option.
> On Mar 2, 2021, at 7:10 PM, Peter Geoghegan <[email protected]> wrote:
>
> I don't think that it's acceptable for your new check to raise a
> WARNING instead of an ERROR.
You already responded to Peter, and I can see that after raising WARNINGs about an index, the code raises an ERROR. That is different from behavior that pg_amcheck currently expects from contrib/amcheck functions. It will be interesting to see if that makes integration harder.
> On Mar 2, 2021, at 6:54 PM, Peter Geoghegan <[email protected]> wrote:
>
>> The regression test you provided is not portable. I am getting lots of errors due to differing output of the form "page lsn=0/4DAD7E0". You might turn this into a TAP test and use a regular expression to check the output.
>
> I would test this using a custom opclass that does simple fault
> injection. For example, an opclass that indexes integers, but can be
> configured to dynamically make 0 values equal or unequal to each
> other. That's more representative of real-world problems.
>
> You "break the warranty" by updating pg_index, even compared to
> updating other system catalogs. In particular, you break the
> "indcheckxmin wait -- wait for xmin to be old before using index"
> stuff in get_relation_info(). So it seems worse than updating
> pg_attribute, for example (which is something that the tests do
> already).
Take a look at src/bin/pg_amcheck/t/005_opclass_damage.pl for an example of changing an opclass to test btree index breakage.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-04-08 14:36 ` David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: David Steele @ 2021-04-08 14:36 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Mark Dilger <[email protected]>
On 3/15/21 11:11 AM, Mark Dilger wrote:
>
>> On Mar 2, 2021, at 6:08 AM, Pavel Borisov <[email protected]> wrote:
>>
>> I completely agree that checking uniqueness requires looking at the heap, but I don't agree that every caller of bt_index_check on an index wants that particular check to be performed. There are multiple ways in which an index might be corrupt, and Peter wrote the code to only check some of them by default, with options to expand the checks to other things. This is why heapallindexed is optional. If you don't want to pay the price of checking all entries in the heap against the btree, you don't have to.
>>
>> I've got the idea and revised the patch accordingly. Thanks!
>> Pfa v4 of a patch. I've added an optional argument to allow uniqueness checks for the unique indexes.
>> Also, I added a test variant to make them work on 32-bit systems. Unfortunately, converting the regression test to TAP would be a pain for me. Hope it can be used now as a 2-variant regression test for 32 and 64 bit systems.
>>
>> Thank you for your consideration!
>>
>> --
>> Best regards,
>> Pavel Borisov
>>
>> Postgres Professional: http://postgrespro.com
>> <v4-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch>
>
> Looking over v4, here are my review comments...
This patch appears to need some work and has not been updated in several
weeks, so marking Returned with Feedback.
Please submit to the next CF when you have a new patch.
Regards,
--
-David
[email protected]
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
@ 2021-12-20 15:37 ` Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2021-12-20 15:37 UTC (permalink / raw)
To: David Steele <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Mark Dilger <[email protected]>; Maxim Orlov <[email protected]>
>
> >> I completely agree that checking uniqueness requires looking at the
> heap, but I don't agree that every caller of bt_index_check on an index
> wants that particular check to be performed. There are multiple ways in
> which an index might be corrupt, and Peter wrote the code to only check
> some of them by default, with options to expand the checks to other
> things. This is why heapallindexed is optional. If you don't want to pay
> the price of checking all entries in the heap against the btree, you don't
> have to.
> >>
> >> I've got the idea and revised the patch accordingly. Thanks!
> >> Pfa v4 of a patch. I've added an optional argument to allow uniqueness
> checks for the unique indexes.
> >> Also, I added a test variant to make them work on 32-bit systems.
> Unfortunately, converting the regression test to TAP would be a pain for
> me. Hope it can be used now as a 2-variant regression test for 32 and 64
> bit systems.
> >>
> >> Thank you for your consideration!
> >>
> >> --
> >> Best regards,
> >> Pavel Borisov
> >>
> >> Postgres Professional: http://postgrespro.com
> >> <v4-0001-Make-amcheck-checking-UNIQUE-constraint-for-btree.patch>
> >
> > Looking over v4, here are my review comments...
>
Mark and Peter, big thanks for your ideas!
I had little time to work on this feature until recently, but finally, I've
elaborated v5 patch (PFA)
It contains the following improvements, most of which are based on your
consideration:
- Amcheck tests are reworked into TAP-tests with "break the warranty" by
comparison function changes in the opclass instead of pg_index update.
Mark, again thanks for the sample!
- Added new --checkunique option into pg_amcheck.
- Added documentation and tests for amcheck and pg_amcheck new functions.
- Results are output at ERROR log level like it is done in the other
amcheck tests.
- Rare case of inability to check due to the first entry on a leaf page
being both: (1) equal to the last one on the previous page and (2) deleted
in the heap, is demoted to DEBUG1 log level. In this, I folowed Peter's
consideration that amcheck should do its best to check, but can not always
verify everything. The case is expected to be extremely rare.
- Made pages connectivity based on btpo_next (corrected a bug in the code,
I found meanwhile)
- If snapshot is already taken in heapallindexed case, reuse it for unique
constraint check.
The patch is pgindented and rebased on the current PG master code.
I'd like to re-attach the patch v5 to the upcoming CF if you don't mind.
I also want to add that some customers face index uniqueness
violations more often than I've expected, and I hope this check could also
help some other PostgreSQL customers.
Your further considerations are welcome as always!
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v5-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch (29.8K, ../../CALT9ZEF=1q-mJiHV6E9e0cQmOYur0vtSX387KLZGs5U3me7vxw@mail.gmail.com/3-v5-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch)
download | inline diff:
From b9bdb2dde7d97e09e9a95daf43c8c843ff3876d8 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 20 Dec 2021 16:57:03 +0400
Subject: [PATCH v5] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 27 ++
contrib/amcheck/sql/check_btree.sql | 7 +
contrib/amcheck/verify_nbtree.c | 327 ++++++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 17 +-
src/bin/pg_amcheck/t/005_opclass_damage.pl | 75 ++++-
9 files changed, 457 insertions(+), 25 deletions(-)
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 5a3f1ef737c..47250ec2f0f 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,11 +177,38 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
+ERROR: table "bttest_unique" does not exist
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..1acee3a2439 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,11 +115,18 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index d3b29d3d890..80ddf3c819b 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -323,7 +351,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/*
@@ -418,7 +446,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -450,6 +479,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -507,6 +538,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -633,6 +681,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -873,6 +923,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1027,6 +1233,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1049,6 +1258,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1439,6 +1655,41 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1456,12 +1707,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1495,6 +1750,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1540,9 +1834,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1709,6 +2005,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 11d1eb5af23..0f23bbd575b 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index d4a53c8e636..7089ed6459f 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -267,6 +269,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -449,6 +452,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
fprintf(stderr,
_("Try \"%s --help\" for more information.\n"),
@@ -871,7 +877,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s, "
+ "checkunique := %s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -880,11 +887,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (opts.checkunique ? "true" : "false"),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s,"
+ "checkunique := %s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -892,6 +901,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (opts.checkunique ? "true" : "false"),
rel->reloid);
}
@@ -1187,6 +1197,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index 2f86f4f2a40..78ecb7c6321 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -8,7 +8,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 5;
+use Test::More tests => 10;
my $node = PostgreSQL::Test::Cluster->new('test');
$node->init;
@@ -22,6 +22,17 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
@@ -30,6 +41,21 @@ $node->safe_psql(
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+
+ CREATE OPERATOR CLASS int4_custom_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
+
+ CREATE TABLE bttest_unique (i int4);
+ INSERT INTO bttest_unique
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON bttest_unique
+ USING btree (i int4_custom_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,3 +83,50 @@ $node->command_checks_all(
[],
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-12-20 18:02 ` Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Mark Dilger @ 2021-12-20 18:02 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: David Steele <[email protected]>; Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>
> On Dec 20, 2021, at 7:37 AM, Pavel Borisov <[email protected]> wrote:
>
> The patch is pgindented and rebased on the current PG master code.
Thank you, Pavel.
The tests in check_btree.sql no longer create a bttest_unique table, so the DROP TABLE is surplusage:
+DROP TABLE bttest_unique;
+ERROR: table "bttest_unique" does not exist
The changes in pg_amcheck.c to pass the new checkunique parameter will likely need to be based on a amcheck version check. The implementation of prepare_btree_command() in pg_amcheck.c should be kept compatible with older versions of amcheck, because it connects to remote servers and you can't know in advance that the remote servers are as up-to-date as the machine where pg_amcheck is installed. I'm thinking specifically about this change:
@@ -871,7 +877,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s, "
+ "checkunique := %s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
If the user calls pg_amcheck with --checkunique, and one or more remote servers have an amcheck version < 1.4, at a minimum you'll need to avoid calling bt_index_parent_check with that parameter, and probably also you'll either need to raise a warning or perhaps an error telling the user that such a check cannot be performed.
You've forgotten to include contrib/amcheck/amcheck--1.3--1.4.sql in the v5 patch, resulting in a failed install:
/usr/bin/install -c -m 644 ./amcheck--1.3--1.4.sql ./amcheck--1.2--1.3.sql ./amcheck--1.1--1.2.sql ./amcheck--1.0--1.1.sql ./amcheck--1.0.sql '/Users/mark.dilger/hydra/unique_review.5/tmp_install/Users/mark.dilger/pgtest/test_install/share/postgresql/extension/'
install: ./amcheck--1.3--1.4.sql: No such file or directory
make[2]: *** [install] Error 71
make[1]: *** [checkprep] Error 2
Using the one from the v4 patch fixes the problem. Please include this file in v6, to simplify the review process.
The changes to t/005_opclass_damage.pl look ok. The creation of a new table for the new test seems unnecessary, but only problematic in that it makes the test slightly longer to read. I recommend changing the test to use the same table that the prior test uses, but that is just a recommendation, not a requirement.
You should add coverage for --checkunique to t/003_check.pl.
You should add coverage for multiple PostgreSQL::Test::Cluster instances running differing versions of amcheck, perhaps some on version 1.3 and some on version 1.4. Then test that the --checkunique option works adequately.
I have not reviewed the changes to verify_nbtree.c. I'll leave that to Peter.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-12-22 08:01 ` Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2021-12-22 08:01 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
>
> The tests in check_btree.sql no longer create a bttest_unique table, so
> the DROP TABLE is surplusage:
>
> +DROP TABLE bttest_unique;
> +ERROR: table "bttest_unique" does not exist
>
>
> The changes in pg_amcheck.c to pass the new checkunique parameter will
> likely need to be based on a amcheck version check. The implementation of
> prepare_btree_command() in pg_amcheck.c should be kept compatible with
> older versions of amcheck, because it connects to remote servers and you
> can't know in advance that the remote servers are as up-to-date as the
> machine where pg_amcheck is installed. I'm thinking specifically about
> this change:
>
> @@ -871,7 +877,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo
> *rel, PGconn *conn)
> if (opts.parent_check)
> appendPQExpBuffer(sql,
> "SELECT %s.bt_index_parent_check("
> - "index := c.oid, heapallindexed := %s,
> rootdescend := %s)"
> + "index := c.oid, heapallindexed := %s,
> rootdescend := %s, "
> + "checkunique := %s)"
> "\nFROM pg_catalog.pg_class c,
> pg_catalog.pg_index i "
> "WHERE c.oid = %u "
> "AND c.oid = i.indexrelid "
>
> If the user calls pg_amcheck with --checkunique, and one or more remote
> servers have an amcheck version < 1.4, at a minimum you'll need to avoid
> calling bt_index_parent_check with that parameter, and probably also you'll
> either need to raise a warning or perhaps an error telling the user that
> such a check cannot be performed.
>
>
> You've forgotten to include contrib/amcheck/amcheck--1.3--1.4.sql in the
> v5 patch, resulting in a failed install:
>
> /usr/bin/install -c -m 644 ./amcheck--1.3--1.4.sql ./amcheck--1.2--1.3.sql
> ./amcheck--1.1--1.2.sql ./amcheck--1.0--1.1.sql ./amcheck--1.0.sql
> '/Users/mark.dilger/hydra/unique_review.5/tmp_install/Users/mark.dilger/pgtest/test_install/share/postgresql/extension/'
> install: ./amcheck--1.3--1.4.sql: No such file or directory
> make[2]: *** [install] Error 71
> make[1]: *** [checkprep] Error 2
>
> Using the one from the v4 patch fixes the problem. Please include this
> file in v6, to simplify the review process.
>
>
> The changes to t/005_opclass_damage.pl look ok. The creation of a new
> table for the new test seems unnecessary, but only problematic in that it
> makes the test slightly longer to read. I recommend changing the test to
> use the same table that the prior test uses, but that is just a
> recommendation, not a requirement.
>
> You should add coverage for --checkunique to t/003_check.pl.
>
> You should add coverage for multiple PostgreSQL::Test::Cluster instances
> running differing versions of amcheck, perhaps some on version 1.3 and some
> on version 1.4. Then test that the --checkunique option works adequately.
>
Thank you, Mark!
In v6 (PFA) I've made the changes on your advice i.e.
- pg_amcheck with --checkunique option will ignore uniqueness check (with a
warning) if amcheck version in a db is <1.4 and doesn't support the feature.
- fixed unnecessary drop table in regression
- use the existing table for uniqueness check in 005_opclass_damage.pl
- added tests into 003_check.pl . It is only smoke test that just verifies
new functions.
- added test contrib/amcheck/t/004_verify_nbtree_unique.pl it is more
extensive test based on opclass damage which was intended to be main test
for amcheck, but which I've forgotten to add to commit in v5.
005_opclass_damage.pl test, which you've seen in v5 is largely based on
first part of 004_verify_nbtree_unique.pl (with the later calling
pg_amcheck, and the former calling bt_index_check(),
bt_index_parent_check() )
- added forgotten upgrade script amcheck--1.3--1.4.sql (from v4)
You are welcome with more considerations.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v6-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch (42.3K, ../../CALT9ZEEx+3NwspWRx2eRRWOG16SYtKSD+7825jAh1wdJux_EkA@mail.gmail.com/3-v6-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch)
download | inline diff:
From cf2d11a2e359d173bb92f6345a7f060630c03d64 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 22 Dec 2021 11:39:15 +0400
Subject: [PATCH v6] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 25 ++
contrib/amcheck/sql/check_btree.sql | 6 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 327 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 56 ++-
src/bin/pg_amcheck/t/003_check.pl | 48 ++-
src/bin/pg_amcheck/t/005_opclass_damage.pl | 75 +++-
12 files changed, 798 insertions(+), 31 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..1caba148aa4
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 5a3f1ef737c..0144767b36e 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,6 +177,31 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..4eb5ffb21d3 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,6 +115,12 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..a99e474f1f2
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 looks equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck get uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck get uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index d3b29d3d890..80ddf3c819b 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -323,7 +351,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/*
@@ -418,7 +446,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -450,6 +479,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -507,6 +538,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -633,6 +681,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -873,6 +923,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1027,6 +1233,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1049,6 +1258,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1439,6 +1655,41 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1456,12 +1707,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1495,6 +1750,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1540,9 +1834,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1709,6 +2005,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 11d1eb5af23..0f23bbd575b 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index d4a53c8e636..30914db55b2 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -449,6 +453,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
fprintf(stderr,
_("Try \"%s --help\" for more information.\n"),
@@ -614,6 +621,34 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ if (opts.checkunique == true)
+ {
+ dat->is_checkunique = true;
+
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = pstrdup(PQgetvalue(result, 0, 1));
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -871,7 +906,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -880,11 +916,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -892,6 +930,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1100,17 +1139,17 @@ verify_btree_slot_handler(PGresult *res, PGconn *conn, void *context)
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
- int ntups = PQntuples(res);
+ int ntups = PQntuples(res);
if (ntups > 1)
{
/*
* We expect the btree checking functions to return one void row
* each, or zero rows if the check was skipped due to the object
- * being in the wrong state to be checked, so we should output some
- * sort of warning if we get anything more, not because it
- * indicates corruption, but because it suggests a mismatch between
- * amcheck and pg_amcheck versions.
+ * being in the wrong state to be checked, so we should output
+ * some sort of warning if we get anything more, not because it
+ * indicates corruption, but because it suggests a mismatch
+ * between amcheck and pg_amcheck versions.
*
* In conjunction with --progress, anything written to stderr at
* this time would present strangely to the user without an extra
@@ -1187,6 +1226,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 5913fcc5305..d3f0259bcbd 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -8,7 +8,7 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Fcntl qw(:seek);
-use Test::More tests => 63;
+use Test::More tests => 75;
my ($node, $port, %corrupt_page, %remove_relation);
@@ -258,6 +258,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,3 +520,46 @@ $node->command_checks_all(
[ @cmd, '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
\ No newline at end of file
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index 2f86f4f2a40..78ecb7c6321 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -8,7 +8,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 5;
+use Test::More tests => 10;
my $node = PostgreSQL::Test::Cluster->new('test');
$node->init;
@@ -22,6 +22,17 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
@@ -30,6 +41,21 @@ $node->safe_psql(
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+
+ CREATE OPERATOR CLASS int4_custom_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
+
+ CREATE TABLE bttest_unique (i int4);
+ INSERT INTO bttest_unique
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON bttest_unique
+ USING btree (i int4_custom_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,3 +83,50 @@ $node->command_checks_all(
[],
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-12-23 16:31 ` Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Mark Dilger @ 2021-12-23 16:31 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
> On Dec 22, 2021, at 12:01 AM, Pavel Borisov <[email protected]> wrote:
>
> Thank you, Mark!
>
> In v6 (PFA) I've made the changes on your advice i.e.
>
> - pg_amcheck with --checkunique option will ignore uniqueness check (with a warning) if amcheck version in a db is <1.4 and doesn't support the feature.
Ok.
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = pstrdup(PQgetvalue(result, 0, 1));
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
The pstrdup is unnecessary but harmless.
> - fixed unnecessary drop table in regression
Ok.
> - use the existing table for uniqueness check in 005_opclass_damage.pl
It appears you still create a new table, bttest_unique, rather than using the existing table int4tbl. That's fine.
> - added tests into 003_check.pl . It is only smoke test that just verifies new functions.
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas');
+
You have borrowed the existing tests but forgot to change their names. (The last line of each check is the test name, such as 'pg_amcheck smoke test --parent-check'.) Please make each test name unique.
> - added test contrib/amcheck/t/004_verify_nbtree_unique.pl it is more extensive test based on opclass damage which was intended to be main test for amcheck, but which I've forgotten to add to commit in v5.
> 005_opclass_damage.pl test, which you've seen in v5 is largely based on first part of 004_verify_nbtree_unique.pl (with the later calling pg_amcheck, and the former calling bt_index_check(), bt_index_parent_check() )
Ok.
> - added forgotten upgrade script amcheck--1.3--1.4.sql (from v4)
Ok.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-12-23 18:05 ` Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Максим Орлов @ 2021-12-23 18:05 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Pavel Borisov <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
>
> The pstrdup is unnecessary but harmless.
>
> > - use the existing table for uniqueness check in 005_opclass_damage.pl
>
> It appears you still create a new table, bttest_unique, rather than using
> the existing table int4tbl. That's fine.
>
> > - added tests into 003_check.pl . It is only smoke test that just
> verifies new functions.
>
> You have borrowed the existing tests but forgot to change their names.
> (The last line of each check is the test name, such as 'pg_amcheck smoke
> test --parent-check'.) Please make each test name unique.
>
Thanks for your review! Fixed all these remaining things from patch v6.
PFA v7 patch.
---
Best regards,
Maxim Orlov.
Attachments:
[application/octet-stream] v7-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch (42.3K, ../../CACG=ezZYRH6u5a642_TC+LidT8UKACN+UU48B07ioJFnZ5aOtA@mail.gmail.com/3-v7-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch)
download | inline diff:
From 10c94688e3e0aae2f3ac94026c04161f7b66b04a Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 8 Feb 2021 12:26:08 +0400
Subject: [PATCH v7] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 25 ++
contrib/amcheck/sql/check_btree.sql | 6 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 327 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 56 ++-
src/bin/pg_amcheck/t/003_check.pl | 48 ++-
src/bin/pg_amcheck/t/005_opclass_damage.pl | 68 +++-
12 files changed, 791 insertions(+), 31 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..1caba148aa4
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 5a3f1ef737c..0144767b36e 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,6 +177,31 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..4eb5ffb21d3 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,6 +115,12 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..a99e474f1f2
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 looks equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck get uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck get uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index d3b29d3d890..80ddf3c819b 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -323,7 +351,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/*
@@ -418,7 +446,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -450,6 +479,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -507,6 +538,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -633,6 +681,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -873,6 +923,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1027,6 +1233,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1049,6 +1258,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1439,6 +1655,41 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque))
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple).
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1456,12 +1707,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1495,6 +1750,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1540,9 +1834,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1709,6 +2005,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 11d1eb5af23..0f23bbd575b 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index d4a53c8e636..c4711d4a956 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -449,6 +453,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
fprintf(stderr,
_("Try \"%s --help\" for more information.\n"),
@@ -614,6 +621,34 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ if (opts.checkunique == true)
+ {
+ dat->is_checkunique = true;
+
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -871,7 +906,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -880,11 +916,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -892,6 +930,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1100,17 +1139,17 @@ verify_btree_slot_handler(PGresult *res, PGconn *conn, void *context)
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
- int ntups = PQntuples(res);
+ int ntups = PQntuples(res);
if (ntups > 1)
{
/*
* We expect the btree checking functions to return one void row
* each, or zero rows if the check was skipped due to the object
- * being in the wrong state to be checked, so we should output some
- * sort of warning if we get anything more, not because it
- * indicates corruption, but because it suggests a mismatch between
- * amcheck and pg_amcheck versions.
+ * being in the wrong state to be checked, so we should output
+ * some sort of warning if we get anything more, not because it
+ * indicates corruption, but because it suggests a mismatch
+ * between amcheck and pg_amcheck versions.
*
* In conjunction with --progress, anything written to stderr at
* this time would present strangely to the user without an extra
@@ -1187,6 +1226,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 5913fcc5305..de125ed3c0c 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -8,7 +8,7 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Fcntl qw(:seek);
-use Test::More tests => 63;
+use Test::More tests => 75;
my ($node, $port, %corrupt_page, %remove_relation);
@@ -258,6 +258,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,3 +520,46 @@ $node->command_checks_all(
[ @cmd, '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index 2f86f4f2a40..5219c6ec0f5 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -8,7 +8,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 5;
+use Test::More tests => 10;
my $node = PostgreSQL::Test::Cluster->new('test');
$node->init;
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,3 +76,50 @@ $node->command_checks_all(
[],
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
--
2.32.0 (Apple Git-132)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
@ 2022-02-21 14:14 ` Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Maxim Orlov @ 2022-02-21 14:14 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
I've updated the patch due to recent changes by Daniel Gustafsson
(549ec201d6132b7).
--
Best regards,
Maxim Orlov.
Attachments:
[application/octet-stream] v10-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.4K, ../../CACG=ezZho9PjoF5M+=hBqtfjur-jBmJ0b=GasQ4Yds9d+DUUgw@mail.gmail.com/3-v10-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From d23722b8f083ecfec69307e24c4748c9ddb09ff0 Mon Sep 17 00:00:00 2001
From: Maxim Orlov <[email protected]>
Date: Mon, 21 Feb 2022 17:03:59 +0300
Subject: [PATCH v10] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 60 +++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 818 insertions(+), 29 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..1caba148aa4
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 5a3f1ef737c..d6d578e9995 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,11 +177,53 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..8e09f43c373 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,11 +115,25 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..a99e474f1f2
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 looks equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck get uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck get uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index d2510ee6480..4b947558225 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -323,7 +351,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/*
@@ -418,7 +446,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -450,6 +479,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -507,6 +538,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -633,6 +681,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -873,6 +923,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1027,6 +1233,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1049,6 +1258,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1439,6 +1655,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1456,12 +1709,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1495,6 +1752,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = (BTPageOpaque) PageGetSpecialPointer(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1540,9 +1836,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1709,6 +2007,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 11d1eb5af23..0f23bbd575b 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 6607f729382..b3d393c500b 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -449,6 +453,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
fprintf(stderr,
_("Try \"%s --help\" for more information.\n"),
@@ -614,6 +621,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check version of amcheck extension. Skip requested unique constraint
+ * check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -871,7 +910,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -880,11 +920,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -892,6 +934,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1100,17 +1143,17 @@ verify_btree_slot_handler(PGresult *res, PGconn *conn, void *context)
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
- int ntups = PQntuples(res);
+ int ntups = PQntuples(res);
if (ntups > 1)
{
/*
* We expect the btree checking functions to return one void row
* each, or zero rows if the check was skipped due to the object
- * being in the wrong state to be checked, so we should output some
- * sort of warning if we get anything more, not because it
- * indicates corruption, but because it suggests a mismatch between
- * amcheck and pg_amcheck versions.
+ * being in the wrong state to be checked, so we should output
+ * some sort of warning if we get anything more, not because it
+ * indicates corruption, but because it suggests a mismatch
+ * between amcheck and pg_amcheck versions.
*
* In conjunction with --progress, anything written to stderr at
* this time would present strangely to the user without an extra
@@ -1187,6 +1230,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index d984eacb24f..eb701cb85e3 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -258,6 +258,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -518,4 +521,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index a5e82082700..dcaa333133a 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -58,4 +77,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.25.1
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
@ 2022-04-03 00:02 ` Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Greg Stark @ 2022-04-03 00:02 UTC (permalink / raw)
To: Maxim Orlov <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
This patch was broken by d16773cdc86210493a2874cb0cf93f3883fcda73 "Add
macros in hash and btree AMs to get the special area of their pages"
If it's really just a few macros it should be easy enough to merge but
it would be good to do a rebase given the number of other commits
since February anyways.
On Mon, 21 Feb 2022 at 09:14, Maxim Orlov <[email protected]> wrote:
>
> I've updated the patch due to recent changes by Daniel Gustafsson (549ec201d6132b7).
>
> --
> Best regards,
> Maxim Orlov.
--
greg
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
@ 2022-04-04 09:18 ` Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2022-04-04 09:18 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: Maxim Orlov <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
>
> This patch was broken by d16773cdc86210493a2874cb0cf93f3883fcda73 "Add
> macros in hash and btree AMs to get the special area of their pages"
>
> If it's really just a few macros it should be easy enough to merge but
> it would be good to do a rebase given the number of other commits
> since February anyways.
>
Rebased, thanks!
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v11-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.3K, ../../CALT9ZEFHWy9SG8BZCqqZMSvuSnRtzRG8ESM5tUxTDNBb-0qy6g@mail.gmail.com/3-v11-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From b2bae648a344bf42b874c41a5d633c949e7609f5 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Mon, 4 Apr 2022 12:27:02 +0400
Subject: [PATCH v11] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 60 +++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 818 insertions(+), 29 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..1caba148aa4
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 5a3f1ef737c..d6d578e9995 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -177,11 +177,53 @@ SELECT bt_index_check('toasty', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 97a3e1a20d5..8e09f43c373 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -115,11 +115,25 @@ INSERT INTO toast_bug SELECT repeat('a', 2200);
-- Should not get false positive report of corruption:
SELECT bt_index_check('toasty', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..a99e474f1f2
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 looks equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck get uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck get uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 70278c4f932..ddd9c6783e4 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -323,7 +351,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/*
@@ -418,7 +446,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -450,6 +479,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -507,6 +538,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -633,6 +681,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -873,6 +923,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1027,6 +1233,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1049,6 +1258,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1439,6 +1655,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1456,12 +1709,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1495,6 +1752,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1540,9 +1836,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1709,6 +2007,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936f..a8a83e7cc26 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 6607f729382..b3d393c500b 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -449,6 +453,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
fprintf(stderr,
_("Try \"%s --help\" for more information.\n"),
@@ -614,6 +621,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check version of amcheck extension. Skip requested unique constraint
+ * check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -871,7 +910,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -880,11 +920,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -892,6 +934,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1100,17 +1143,17 @@ verify_btree_slot_handler(PGresult *res, PGconn *conn, void *context)
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
- int ntups = PQntuples(res);
+ int ntups = PQntuples(res);
if (ntups > 1)
{
/*
* We expect the btree checking functions to return one void row
* each, or zero rows if the check was skipped due to the object
- * being in the wrong state to be checked, so we should output some
- * sort of warning if we get anything more, not because it
- * indicates corruption, but because it suggests a mismatch between
- * amcheck and pg_amcheck versions.
+ * being in the wrong state to be checked, so we should output
+ * some sort of warning if we get anything more, not because it
+ * indicates corruption, but because it suggests a mismatch
+ * between amcheck and pg_amcheck versions.
*
* In conjunction with --progress, anything written to stderr at
* this time would present strangely to the user without an extra
@@ -1187,6 +1230,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6b..19a269c1b83 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index a5e82082700..dcaa333133a 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -58,4 +77,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2022-05-11 13:04 ` Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Pavel Borisov @ 2022-05-11 13:04 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: Maxim Orlov <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
v11 patch do not apply due to recent code changes.
Rebased. PFA v12.
Please feel free to check and discuss it.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/octet-stream] v12-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.5K, ../../CALT9ZEG9Zk5TxuDTnZPGe9H+b7DL40gXfeouWCaZ3A70yvyWLw@mail.gmail.com/3-v12-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From 097dbe8593d2b7d4229247c9e65e2136561c9fe8 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v12] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 60 +++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 818 insertions(+), 29 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..1caba148aa4
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f4..9e257ac3bb2 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d05..5afe7f369d7 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..a99e474f1f2
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 looks equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck get uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck get uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index a8791000f87..470db0698c7 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -344,7 +372,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -445,7 +473,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -477,6 +506,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -534,6 +565,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -660,6 +708,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -900,6 +950,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1054,6 +1260,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1076,6 +1285,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1466,6 +1682,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1483,12 +1736,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1522,6 +1779,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1567,9 +1863,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1736,6 +2034,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936f..a8a83e7cc26 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 48cee8c1c4e..3dfd7e918df 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check version of amcheck extension. Skip requested unique constraint
+ * check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1074,17 +1117,17 @@ verify_btree_slot_handler(PGresult *res, PGconn *conn, void *context)
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
- int ntups = PQntuples(res);
+ int ntups = PQntuples(res);
if (ntups > 1)
{
/*
* We expect the btree checking functions to return one void row
* each, or zero rows if the check was skipped due to the object
- * being in the wrong state to be checked, so we should output some
- * sort of warning if we get anything more, not because it
- * indicates corruption, but because it suggests a mismatch between
- * amcheck and pg_amcheck versions.
+ * being in the wrong state to be checked, so we should output
+ * some sort of warning if we get anything more, not because it
+ * indicates corruption, but because it suggests a mismatch
+ * between amcheck and pg_amcheck versions.
*
* In conjunction with --progress, anything written to stderr at
* this time would present strangely to the user without an extra
@@ -1161,6 +1204,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6b..19a269c1b83 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index a5e82082700..dcaa333133a 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -58,4 +77,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2022-05-20 13:46 ` Pavel Borisov <[email protected]>
2022-07-20 14:15 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Pavel Borisov @ 2022-05-20 13:46 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: Maxim Orlov <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
CFbot says v12 patch does not apply.
Rebased. PFA v13.
Your reviews are very much welcome!
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
Attachments:
[application/x-patch] v13-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (42.5K, ../../CALT9ZEG1YAfPBH-ZMnK-cxu0SDdrV_w0QEBs9BLHfWvddpOHFA@mail.gmail.com/3-v13-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From 21fe45c0ae8479e0733bd8caeb5d2a19d715e0d9 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v13] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
With 'checkunique' option bt_index_check() and bt_index_parent_check()
for btree indexes that has unique constraint will check it i.e.
will check that only one heap entry for all equal keys in the index
(including posting list entries) is visible. Report error if not.
pg_amcheck called with --checkunique option will do the same for
all indexes it checks
Authors:
Anastasia Lubennikova <[email protected]>
Pavel Borisov <[email protected]>
Maxim Orlov <[email protected]>
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 50 ++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 813 insertions(+), 24 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..1caba148aa4
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- Don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f4..9e257ac3bb2 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', false, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d05..5afe7f369d7 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', true, true);
+SELECT bt_index_check('bttest_b_idx', false, true);
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
+
+-- Check null values in unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', true, true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', true, true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..a99e474f1f2
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 looks equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck get uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck get uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index a8791000f87..470db0698c7 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -344,7 +372,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -445,7 +473,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -477,6 +506,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -534,6 +565,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot it to check uniqueness of the index For better
+ * performance, take it once per index check. If snapshot already taken,
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -660,6 +708,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -900,6 +950,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced from nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print error message for unique constrain violation in the btree
+ * index under WARNING level and set flag to report ERROR at the end of check
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. If TID of any posting list entry is
+ * visible, and lVis_tid is already valid report duplicate.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique violation between the
+ * posting list entries of a first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible, save info about
+ * it for next comparisons in the loop in bt_page_check(). If also
+ * lVis_tid is already valid, report duplicate.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1054,6 +1260,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint check that only one of table entries for
+ * equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1076,6 +1285,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1466,6 +1682,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique, verify entries uniqueness by checking heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1483,12 +1736,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1522,6 +1779,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint check that not more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* First key on next page is same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1567,9 +1863,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1736,6 +2034,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936f..a8a83e7cc26 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c04655..61dacf1ee44 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index f0b818e987a..3dfd7e918df 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check version of amcheck extension. Skip requested unique constraint
+ * check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1161,6 +1204,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6b..19a269c1b83 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index ce376f239cf..81d392a34e6 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2022-07-20 14:15 ` Aleksander Alekseev <[email protected]>
2022-09-07 13:44 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Dmitry Koval <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Aleksander Alekseev @ 2022-07-20 14:15 UTC (permalink / raw)
To: Postgres hackers <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Mark Dilger <[email protected]>; Peter Geoghegan <[email protected]>
Hi Pavel,
> Rebased. PFA v13.
> Your reviews are very much welcome!
I noticed that this patch is in "Needs Review" state and it has been
stuck for some time now, so I decided to take a look.
```
+SELECT bt_index_parent_check('bttest_a_idx', true, true, true);
+SELECT bt_index_parent_check('bttest_b_idx', true, false, true);
``
1. This "true, false, true" sequence is difficult to read. I suggest
we use named arguments here.
2. I believe there are some minor issues with the comments. E.g. instead of:
- First key on next page is same
- Make values 768 and 769 looks equal
I would write:
- The first key on the next page is the same
- Make values 768 and 769 look equal
There are many little errors like these.
```
+# Copyright (c) 2021, PostgreSQL Global Development Group
```
3. Oh no. The copyright has expired!
```
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
```
4. This piece of documentation was copy-pasted between two functions
without change of the function name.
Other than that to me the patch looks in pretty good shape. Here is
v14 where I fixed my own nitpicks, with the permission of Pavel given
offlist.
If no one sees any other defects I'm going to change the status of the
patch to "Ready to Committer" in a short time.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v14-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.3K, ../../CAJ7c6TP+Optg9TMZJd9+ic9R_c8RLU8DcUCyCoDnZZobb_1wQg@mail.gmail.com/2-v14-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From 37553352e796e877329e111b8744844ec5fb64ee Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v14] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
Add 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes. Only one heap entry for all equal keys in
the index should be visible (including posting list entries). Report an error
otherwise.
pg_amcheck called with --checkunique option will do the same check for all
the indexes it checks.
Author: Anastasia Lubennikova <[email protected]>
Author: Pavel Borisov <[email protected]>
Author: Maxim Orlov <[email protected]>
Reviewed-by: Mark Dilger <[email protected]>
Reviewed-by: Zhihong Yu <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Aleksander Alekseev <[email protected]>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 234 +++++++++++++
contrib/amcheck/verify_nbtree.c | 330 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 50 ++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 814 insertions(+), 24 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..88271687a3 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 0000000000..75574eaa64
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- We don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f75..e67ace01c9 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f..86b38d93f4 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d0..aa461f7fb9 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 0000000000..1df106a232
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,234 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 look equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck finds the uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck finds the uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication is possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 2beeebb163..c14bf62ad4 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -344,7 +372,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -445,7 +473,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -477,6 +506,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -534,6 +565,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot to check the uniqueness of the index. For better
+ * performance take it once per index check. If snapshot already taken
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -660,6 +708,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -900,6 +950,163 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced by nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare and print an error message for unique constrain violation in
+ * a btree index under WARNING level. Also set a flag to report ERROR
+ * at the end of the check.
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. Report duplicate if TID of any posting
+ * list entry is visible and lVis_tid is valid.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique constraint violation between
+ * the posting list entries of the first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible save info about
+ * it for the next comparisons in the loop in bt_page_check(). Report
+ * duplicate if lVis_tid is already valid.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1054,6 +1261,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint make sure that only one of table entries
+ * for equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1076,6 +1286,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1466,6 +1683,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique verify entries uniqueness by checking the heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1483,12 +1737,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1522,6 +1780,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint make sure that no more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* The first key on the next page is the same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1567,9 +1864,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1736,6 +2035,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936..b6f3adc612 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_parent_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c0465..61dacf1ee4 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 3cff319f02..3ad5927319 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check the version of amcheck extension. Skip requested unique
+ * constraint check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1163,6 +1206,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6..19a269c1b8 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index ce376f239c..81d392a34e 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.36.1
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-07-20 14:15 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
@ 2022-09-07 13:44 ` Dmitry Koval <[email protected]>
2022-09-08 13:29 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Karina Litskevich <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Dmitry Koval @ 2022-09-07 13:44 UTC (permalink / raw)
To: Postgres hackers <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Mark Dilger <[email protected]>; Peter Geoghegan <[email protected]>; Aleksander Alekseev <[email protected]>
Hi!
I would make two cosmetic changes.
1. I suggest replace description of function bt_report_duplicate() from
```
/*
* Prepare and print an error message for unique constrain violation in
* a btree index under WARNING level. Also set a flag to report ERROR
* at the end of the check.
*/
```
to
```
/*
* Prepare an error message for unique constrain violation in
* a btree index and report ERROR.
*/
```
2. I think will be better to change test 004_verify_nbtree_unique.pl -
replace
```
use Test::More tests => 6;
```
to
```
use Test::More;
...
done_testing();
```
(same as in the other three tests).
--
With best regards,
Dmitry Koval
Postgres Professional: http://postgrespro.com
From 93a10abd0afb14b264e4cf59f7e92f619dd9b11a Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v15] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
Add 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes. Only one heap entry for all equal keys in
the index should be visible (including posting list entries). Report an error
otherwise.
pg_amcheck called with --checkunique option will do the same check for all
the indexes it checks.
Author: Anastasia Lubennikova <[email protected]>
Author: Pavel Borisov <[email protected]>
Author: Maxim Orlov <[email protected]>
Reviewed-by: Mark Dilger <[email protected]>
Reviewed-by: Zhihong Yu <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Aleksander Alekseev <[email protected]>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 235 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 50 ++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 814 insertions(+), 24 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..88271687a3 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 0000000000..75574eaa64
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- We don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f75..e67ace01c9 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f..86b38d93f4 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d0..aa461f7fb9 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 0000000000..83572959bd
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,235 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 look equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck finds the uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck finds the uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication is possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
+done_testing();
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 2beeebb163..169a16b82c 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -344,7 +372,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -445,7 +473,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -477,6 +506,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -534,6 +565,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot to check the uniqueness of the index. For better
+ * performance take it once per index check. If snapshot already taken
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -660,6 +708,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -900,6 +950,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced by nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare an error message for unique constrain violation in
+ * a btree index and report ERROR.
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. Report duplicate if TID of any posting
+ * list entry is visible and lVis_tid is valid.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique constraint violation between
+ * the posting list entries of the first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible save info about
+ * it for the next comparisons in the loop in bt_page_check(). Report
+ * duplicate if lVis_tid is already valid.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1054,6 +1260,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint make sure that only one of table entries
+ * for equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1076,6 +1285,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1466,6 +1682,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique verify entries uniqueness by checking the heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1483,12 +1736,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1522,6 +1779,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint make sure that no more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* The first key on the next page is the same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1567,9 +1863,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1736,6 +2034,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936..b6f3adc612 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_parent_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c0465..61dacf1ee4 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index fea35e4b14..956fb6f565 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check the version of amcheck extension. Skip requested unique
+ * constraint check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1163,6 +1206,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6..19a269c1b8 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index ce376f239c..81d392a34e 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.31.0.windows.1
Attachments:
[text/plain] v15-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.3K, ../../[email protected]/2-v15-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From 93a10abd0afb14b264e4cf59f7e92f619dd9b11a Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v15] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
Add 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes. Only one heap entry for all equal keys in
the index should be visible (including posting list entries). Report an error
otherwise.
pg_amcheck called with --checkunique option will do the same check for all
the indexes it checks.
Author: Anastasia Lubennikova <[email protected]>
Author: Pavel Borisov <[email protected]>
Author: Maxim Orlov <[email protected]>
Reviewed-by: Mark Dilger <[email protected]>
Reviewed-by: Zhihong Yu <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Aleksander Alekseev <[email protected]>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 235 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 50 ++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 814 insertions(+), 24 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..88271687a3 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 0000000000..75574eaa64
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- We don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f75..e67ace01c9 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f..86b38d93f4 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d0..aa461f7fb9 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 0000000000..83572959bd
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,235 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 look equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck finds the uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck finds the uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication is possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
+done_testing();
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 2beeebb163..169a16b82c 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -344,7 +372,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -445,7 +473,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -477,6 +506,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -534,6 +565,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot to check the uniqueness of the index. For better
+ * performance take it once per index check. If snapshot already taken
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -660,6 +708,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -900,6 +950,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced by nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare an error message for unique constrain violation in
+ * a btree index and report ERROR.
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. Report duplicate if TID of any posting
+ * list entry is visible and lVis_tid is valid.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique constraint violation between
+ * the posting list entries of the first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible save info about
+ * it for the next comparisons in the loop in bt_page_check(). Report
+ * duplicate if lVis_tid is already valid.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1054,6 +1260,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint make sure that only one of table entries
+ * for equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1076,6 +1285,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1466,6 +1682,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique verify entries uniqueness by checking the heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1483,12 +1736,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1522,6 +1779,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint make sure that no more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* The first key on the next page is the same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1567,9 +1863,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1736,6 +2034,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936..b6f3adc612 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_parent_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c0465..61dacf1ee4 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index fea35e4b14..956fb6f565 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
static const char *progname = NULL;
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check the version of amcheck extension. Skip requested unique
+ * constraint check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1163,6 +1206,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6..19a269c1b8 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index ce376f239c..81d392a34e 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.31.0.windows.1
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-07-20 14:15 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2022-09-07 13:44 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Dmitry Koval <[email protected]>
@ 2022-09-08 13:29 ` Karina Litskevich <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Karina Litskevich @ 2022-09-08 13:29 UTC (permalink / raw)
To: Postgres hackers <[email protected]>; Pavel Borisov <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Mark Dilger <[email protected]>; Peter Geoghegan <[email protected]>; Aleksander Alekseev <[email protected]>; Dmitry Koval <[email protected]>
Hi,
I also would like to suggest a cosmetic change.
In v15 a new field checkunique is added after heapallindexed and before
no_btree_expansion fields in struct definition, but in initialisation it is
added after no_btree_expansion:
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,7 +133,8 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .no_btree_expansion = false
+ .no_btree_expansion = false,
+ .checkunique = false
};
I suggest to add checkunique field between heapallindexed and
no_btree_expansion fields in initialisation as well as in definition:
@@ -132,6 +133,7 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
+ .checkunique = false,
.no_btree_expansion = false
};
--
Best regards,
Litskevich Karina
Postgres Professional: http://postgrespro.com/
Attachments:
[text/x-patch] v16-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.2K, ../../CACiT8iYi5HKksUenN2D+BxUGKvpRxuVdR37PWGwvDDea2OO6Tg@mail.gmail.com/3-v16-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From 56fe2b608b46c6c97900bbb63b2169e2997bc8cc Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v16] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
Add 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes. Only one heap entry for all equal keys in
the index should be visible (including posting list entries). Report an error
otherwise.
pg_amcheck called with --checkunique option will do the same check for all
the indexes it checks.
Author: Anastasia Lubennikova <[email protected]>
Author: Pavel Borisov <[email protected]>
Author: Maxim Orlov <[email protected]>
Reviewed-by: Mark Dilger <[email protected]>
Reviewed-by: Zhihong Yu <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Aleksander Alekseev <[email protected]>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 235 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 48 ++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
12 files changed, 813 insertions(+), 23 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..88271687a3 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 0000000000..75574eaa64
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- We don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f75..e67ace01c9 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f..86b38d93f4 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d0..aa461f7fb9 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 0000000000..83572959bd
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,235 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 look equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck finds the uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck finds the uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication is possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
+done_testing();
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 2beeebb163..169a16b82c 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -79,11 +79,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -138,19 +146,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -190,7 +212,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -203,17 +225,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -227,13 +252,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -243,7 +271,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -344,7 +372,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -445,7 +473,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -477,6 +506,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -534,6 +565,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot to check the uniqueness of the index. For better
+ * performance take it once per index check. If snapshot already taken
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -660,6 +708,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -900,6 +950,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced by nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare an error message for unique constrain violation in
+ * a btree index and report ERROR.
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. Report duplicate if TID of any posting
+ * list entry is visible and lVis_tid is valid.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique constraint violation between
+ * the posting list entries of the first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible save info about
+ * it for the next comparisons in the loop in bt_page_check(). Report
+ * duplicate if lVis_tid is already valid.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1054,6 +1260,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint make sure that only one of table entries
+ * for equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1076,6 +1285,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1466,6 +1682,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique verify entries uniqueness by checking the heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1483,12 +1736,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1522,6 +1779,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint make sure that no more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* The first key on the next page is the same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1567,9 +1863,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1736,6 +2034,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936..b6f3adc612 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_parent_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c0465..61dacf1ee4 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index fea35e4b14..ac9ce2a341 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,6 +133,7 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
+ .checkunique = false,
.no_btree_expansion = false
};
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check the version of amcheck extension. Skip requested unique
+ * constraint check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1163,6 +1206,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6..19a269c1b8 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index ce376f239c..81d392a34e 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.25.1
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2022-09-22 15:13 ` Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-09-22 15:13 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Greg Stark <[email protected]>; Maxim Orlov <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
Hi,
On 2022-05-20 17:46:38 +0400, Pavel Borisov wrote:
> CFbot says v12 patch does not apply.
> Rebased. PFA v13.
> Your reviews are very much welcome!
Due to the merge of the meson based build this patch needs to be
adjusted: https://cirrus-ci.com/build/6350479973154816
Looks like you need to add amcheck--1.3--1.4.sql to the list of files to be
installed and t/004_verify_nbtree_unique.pl to the tests.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
@ 2022-09-27 08:04 ` Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Maxim Orlov @ 2022-09-27 08:04 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
On Thu, 22 Sept 2022 at 18:13, Andres Freund <[email protected]> wrote:
> Due to the merge of the meson based build this patch needs to be
> adjusted: https://cirrus-ci.com/build/6350479973154816
>
> Looks like you need to add amcheck--1.3--1.4.sql to the list of files to be
> installed and t/004_verify_nbtree_unique.pl to the tests.
>
> Greetings,
>
> Andres Freund
>
Thanks! Fixed.
--
Best regards,
Maxim Orlov.
Attachments:
[application/octet-stream] v17-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch (43.8K, ../../CACG=ezbRVZrdTguWphiFP-L2cCi_jy+spZA0wPJWA806B0=2dQ@mail.gmail.com/3-v17-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-u.patch)
download | inline diff:
From b7989f959429087328f7b1e521072567154d1167 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Wed, 11 May 2022 15:54:13 +0400
Subject: [PATCH v17] Add option for amcheck and pg_amcheck to check unique
constraint for btree indexes.
Add 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes. Only one heap entry for all equal keys in
the index should be visible (including posting list entries). Report an error
otherwise.
pg_amcheck called with --checkunique option will do the same check for all
the indexes it checks.
Author: Anastasia Lubennikova <[email protected]>
Author: Pavel Borisov <[email protected]>
Author: Maxim Orlov <[email protected]>
Reviewed-by: Mark Dilger <[email protected]>
Reviewed-by: Zhihong Yu <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Aleksander Alekseev <[email protected]>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/meson.build | 2 +
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 235 +++++++++++++
contrib/amcheck/verify_nbtree.c | 329 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 48 ++-
src/bin/pg_amcheck/t/003_check.pl | 45 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
13 files changed, 815 insertions(+), 23 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..88271687a3 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 0000000000..75574eaa64
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- We don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f75..e67ace01c9 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f..86b38d93f4 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/meson.build b/contrib/amcheck/meson.build
index 1db3d20349..1a550ff762 100644
--- a/contrib/amcheck/meson.build
+++ b/contrib/amcheck/meson.build
@@ -12,6 +12,7 @@ install_data(
'amcheck--1.0--1.1.sql',
'amcheck--1.1--1.2.sql',
'amcheck--1.2--1.3.sql',
+ 'amcheck--1.3--1.4.sql',
kwargs: contrib_data_args,
)
@@ -31,6 +32,7 @@ tests += {
't/001_verify_heapam.pl',
't/002_cic.pl',
't/003_cic_2pc.pl',
+ 't/004_verify_nbtree_unique.pl',
],
},
}
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d0..aa461f7fb9 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 0000000000..83572959bd
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,235 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 look equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck finds the uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck finds the uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication is possible only for same values, but
+# with different visibility.
+$node->safe_psql('postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql('postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql('postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok($stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
+done_testing();
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 9021d156eb..cc75697798 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -80,11 +80,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -139,19 +147,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -191,7 +213,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -204,17 +226,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -228,13 +253,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -244,7 +272,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -345,7 +373,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -446,7 +474,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -478,6 +507,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -535,6 +566,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot to check the uniqueness of the index. For better
+ * performance take it once per index check. If snapshot already taken
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -661,6 +709,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -901,6 +951,162 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced by nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare an error message for unique constrain violation in
+ * a btree index and report ERROR.
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\": "
+ "Index %s%s and%s%s "
+ "(point to heap %s and %s) page lsn=%X/%X.",
+ RelationGetRelationName(state->rel),
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i, ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset, BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. Report duplicate if TID of any posting
+ * list entry is visible and lVis_tid is valid.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique constraint violation between
+ * the posting list entries of the first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible save info about
+ * it for the next comparisons in the loop in bt_page_check(). Report
+ * duplicate if lVis_tid is already valid.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) "
+ "in index \"%s\". It doesn't have visible heap tids and key "
+ "is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)). "
+ "Vacuum the table and repeat the check.",
+ targetblock, offset,
+ RelationGetRelationName(state->rel),
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid))));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1055,6 +1261,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint make sure that only one of table entries
+ * for equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1077,6 +1286,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1467,6 +1683,43 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique verify entries uniqueness by checking the heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset, &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique && P_ISLEAF(topaque) &&
+ OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1484,12 +1737,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1523,6 +1780,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint make sure that no more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later
+ */
+ rightkey->scantid = NULL;
+
+ /* The first key on the next page is the same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1568,9 +1864,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1737,6 +2035,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 5d61a33936..b6f3adc612 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_parent_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index cfef6c0465..61dacf1ee4 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 9ce4b11f1e..6c27f857d1 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,6 +133,7 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
+ .checkunique = false,
.no_btree_expansion = false
};
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check the version of amcheck extension. Skip requested unique
+ * constraint check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1163,6 +1206,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 0cf67065d6..19a269c1b8 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,46 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2', '-d', 'db3', '-S', 's*' ],
+ 0, [$no_output_re], [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index ce376f239c..81d392a34e 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.37.0 (Apple Git-136)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
@ 2022-09-28 08:36 ` Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Maxim Orlov @ 2022-09-28 08:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Postgres hackers <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
Hi!
I think, this patch was marked as "Waiting on Author", probably, by
mistake. Since recent changes were done without any significant code
changes and CF bot how happy again.
I'm going to move it to RfC, could I? If not, please tell why.
--
Best regards,
Maxim Orlov.
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
@ 2022-09-28 08:43 ` Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Aleksander Alekseev @ 2022-09-28 08:43 UTC (permalink / raw)
To: Postgres hackers <[email protected]>; +Cc: Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Pavel Borisov <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Hamid Akhtar <[email protected]>
Hi hackers,
> I think, this patch was marked as "Waiting on Author", probably, by mistake. Since recent changes were done without any significant code changes and CF bot how happy again.
>
> I'm going to move it to RfC, could I? If not, please tell why.
I restored the "Ready for Committer" state. I don't think it's a good
practice to change the state every time the patch has a slight
conflict or something. This is not helpful at all. Such things happen
quite regularly and typically are fixed in a couple of days.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
@ 2023-10-24 20:13 ` Alexander Korotkov <[email protected]>
2023-10-30 07:29 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Alexander Korotkov @ 2023-10-24 20:13 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Pavel Borisov <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Hamid Akhtar <[email protected]>
On Wed, Sep 28, 2022 at 11:44 AM Aleksander Alekseev
<[email protected]> wrote:
> > I think, this patch was marked as "Waiting on Author", probably, by mistake. Since recent changes were done without any significant code changes and CF bot how happy again.
> >
> > I'm going to move it to RfC, could I? If not, please tell why.
>
> I restored the "Ready for Committer" state. I don't think it's a good
> practice to change the state every time the patch has a slight
> conflict or something. This is not helpful at all. Such things happen
> quite regularly and typically are fixed in a couple of days.
This patch seems useful to me. I went through the thread, it seems
that all the critics are addressed.
I've rebased this patch. Also, I've run perltidy for tests, split
long errmsg() into errmsg(), errdetail() and errhint(), and do other
minor enchantments.
I think this patch is ready to go. I'm going to push it if there are
no objections.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Teach-contrib-amcheck-to-check-the-unique-constr-v18.patch (43.9K, ../../CAPpHfduqJ9QCOrXDjkPxPRxJRws1FgFELwB2Yz3L6-n7FQMsrA@mail.gmail.com/2-0001-Teach-contrib-amcheck-to-check-the-unique-constr-v18.patch)
download | inline diff:
From a405bdcbca38a9f44da70a20312a9bc81e50c76c Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Tue, 24 Oct 2023 22:07:30 +0300
Subject: [PATCH] Teach contrib/amcheck to check the unique constraint
violation
Add the 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes. Only one heap entry for all equal keys in
the index should be visible (including posting list entries). Report an error
otherwise.
pg_amcheck called with the --checkunique option will do the same check for all
the indexes it checks.
Author: Anastasia Lubennikova <[email protected]>
Author: Pavel Borisov <[email protected]>
Author: Maxim Orlov <[email protected]>
Reviewed-by: Mark Dilger <[email protected]>
Reviewed-by: Zhihong Yu <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Aleksander Alekseev <[email protected]>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/amcheck--1.3--1.4.sql | 29 ++
contrib/amcheck/amcheck.control | 2 +-
contrib/amcheck/expected/check_btree.out | 42 +++
contrib/amcheck/meson.build | 2 +
contrib/amcheck/sql/check_btree.sql | 14 +
contrib/amcheck/t/004_verify_nbtree_unique.pl | 244 +++++++++++++
contrib/amcheck/verify_nbtree.c | 330 +++++++++++++++++-
doc/src/sgml/amcheck.sgml | 14 +-
doc/src/sgml/ref/pg_amcheck.sgml | 11 +
src/bin/pg_amcheck/pg_amcheck.c | 48 ++-
src/bin/pg_amcheck/t/003_check.pl | 50 +++
src/bin/pg_amcheck/t/005_opclass_damage.pl | 65 ++++
13 files changed, 830 insertions(+), 23 deletions(-)
create mode 100644 contrib/amcheck/amcheck--1.3--1.4.sql
create mode 100644 contrib/amcheck/t/004_verify_nbtree_unique.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50b..88271687a3e 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -7,7 +7,7 @@ OBJS = \
verify_nbtree.o
EXTENSION = amcheck
-DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
+DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
new file mode 100644
index 00000000000..75574eaa64b
--- /dev/null
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -0,0 +1,29 @@
+/* contrib/amcheck/amcheck--1.3--1.4.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.4'" to load this file. \quit
+
+-- In order to avoid issues with dependencies when updating amcheck to 1.4,
+-- create new, overloaded versions of the 1.2 bt_index_parent_check signature,
+-- and 1.1 bt_index_check signature.
+
+--
+-- bt_index_parent_check()
+--
+CREATE FUNCTION bt_index_parent_check(index regclass,
+ heapallindexed boolean, rootdescend boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_parent_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+--
+-- bt_index_check()
+--
+CREATE FUNCTION bt_index_check(index regclass,
+ heapallindexed boolean, checkunique boolean)
+RETURNS VOID
+AS 'MODULE_PATHNAME', 'bt_index_check'
+LANGUAGE C STRICT PARALLEL RESTRICTED;
+
+-- We don't want this to be available to public
+REVOKE ALL ON FUNCTION bt_index_parent_check(regclass, boolean, boolean, boolean) FROM PUBLIC;
+REVOKE ALL ON FUNCTION bt_index_check(regclass, boolean, boolean) FROM PUBLIC;
diff --git a/contrib/amcheck/amcheck.control b/contrib/amcheck/amcheck.control
index ab50931f754..e67ace01c99 100644
--- a/contrib/amcheck/amcheck.control
+++ b/contrib/amcheck/amcheck.control
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
-default_version = '1.3'
+default_version = '1.4'
module_pathname = '$libdir/amcheck'
relocatable = true
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index 38791bbc1f4..86b38d93f41 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -199,6 +199,47 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+ bt_index_parent_check
+-----------------------
+
+(1 row)
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+ bt_index_check
+----------------
+
+(1 row)
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -206,5 +247,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/meson.build b/contrib/amcheck/meson.build
index 5b55cf343a9..4c8e2e2f13c 100644
--- a/contrib/amcheck/meson.build
+++ b/contrib/amcheck/meson.build
@@ -23,6 +23,7 @@ install_data(
'amcheck--1.0--1.1.sql',
'amcheck--1.1--1.2.sql',
'amcheck--1.2--1.3.sql',
+ 'amcheck--1.3--1.4.sql',
kwargs: contrib_data_args,
)
@@ -42,6 +43,7 @@ tests += {
't/001_verify_heapam.pl',
't/002_cic.pl',
't/003_cic_2pc.pl',
+ 't/004_verify_nbtree_unique.pl',
],
},
}
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 033c04b4d05..aa461f7fb97 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -135,6 +135,19 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
+-- UNIQUE constraint check
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+
+-- Check that null values in an unique index are not treated as equal
+CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
+INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+CREATE INDEX on bttest_unique_nulls (b,c);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
@@ -142,5 +155,6 @@ DROP TABLE bttest_multi;
DROP TABLE delete_test_table;
DROP TABLE toast_bug;
DROP FUNCTION ifun(int8);
+DROP TABLE bttest_unique_nulls;
DROP OWNED BY regress_bttest_role; -- permissions
DROP ROLE regress_bttest_role;
diff --git a/contrib/amcheck/t/004_verify_nbtree_unique.pl b/contrib/amcheck/t/004_verify_nbtree_unique.pl
new file mode 100644
index 00000000000..b999ab9c176
--- /dev/null
+++ b/contrib/amcheck/t/004_verify_nbtree_unique.pl
@@ -0,0 +1,244 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+# This regression test checks the behavior of the btree validation in the
+# presence of breaking sort order changes.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('test');
+$node->init;
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->start;
+
+# Create a custom operator class and an index which uses it.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE EXTENSION amcheck;
+
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ ---
+ --- Check 1: uniqueness violation.
+ ---
+ CREATE FUNCTION ok_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ ---
+ --- Make values 768 and 769 look equal.
+ ---
+ CREATE FUNCTION bad_cmp1 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 2: uniqueness violation without deduplication.
+ ---
+ CREATE FUNCTION ok_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp2 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 = $2 AND $1 = 400 THEN -1
+ ELSE ok_cmp($1, $2)
+ END;
+ $$;
+
+ ---
+ --- Check 3: uniqueness violation with deduplication.
+ ---
+ CREATE FUNCTION ok_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT ok_cmp($1, $2);
+ $$;
+
+ CREATE FUNCTION bad_cmp3 (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT bad_cmp2($1, $2);
+ $$;
+
+ ---
+ --- Create data.
+ ---
+ CREATE TABLE bttest_unique1 (i int4);
+ INSERT INTO bttest_unique1
+ (SELECT * FROM generate_series(1, 1024) gs);
+
+ CREATE TABLE bttest_unique2 (i int4);
+ INSERT INTO bttest_unique2(i)
+ (SELECT * FROM generate_series(1, 400) gs);
+ INSERT INTO bttest_unique2
+ (SELECT * FROM generate_series(400, 1024) gs);
+
+ CREATE TABLE bttest_unique3 (i int4);
+ INSERT INTO bttest_unique3
+ SELECT * FROM bttest_unique2;
+
+ CREATE OPERATOR CLASS int4_custom_ops1 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp1(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops2 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp2(int4, int4);
+ CREATE OPERATOR CLASS int4_custom_ops3 FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 bad_cmp3(int4, int4);
+
+ CREATE UNIQUE INDEX bttest_unique_idx1
+ ON bttest_unique1
+ USING btree (i int4_custom_ops1)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx2
+ ON bttest_unique2
+ USING btree (i int4_custom_ops2)
+ WITH (deduplicate_items = off);
+ CREATE UNIQUE INDEX bttest_unique_idx3
+ ON bttest_unique3
+ USING btree (i int4_custom_ops3)
+ WITH (deduplicate_items = on);
+));
+
+my ($result, $stdout, $stderr);
+
+#
+# Test 1.
+# - insert seq values
+# - create unique index
+# - break cmp function
+# - amcheck finds the uniqueness violation
+#
+
+# We have not yet broken the index, so we should get no corruption
+$result = $node->safe_psql(
+ 'postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+is($result, '', 'run amcheck on non-broken bttest_unique_idx1');
+
+# Change the operator class to use a function which considers certain different
+# values to be equal.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'bad_cmp1'::regproc
+ WHERE amproc = 'ok_cmp1'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql(
+ 'postgres', q(
+ SELECT bt_index_check('bttest_unique_idx1', true, true);
+));
+ok( $stderr =~ /index uniqueness is violated for index "bttest_unique_idx1"/,
+ 'detected uniqueness violation for index "bttest_unique_idx1"');
+
+#
+# Test 2.
+# - break cmp function
+# - insert seq values with duplicates
+# - create unique index
+# - make cmp function correct
+# - amcheck finds the uniqueness violation
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql(
+ 'postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok( $stderr =~ /item order invariant violated for index "bttest_unique_idx2"/,
+ 'detected item order invariant violation for index "bttest_unique_idx2"');
+
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp2'::regproc
+ WHERE amproc = 'bad_cmp2'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql(
+ 'postgres', q(
+ SELECT bt_index_check('bttest_unique_idx2', true, true);
+));
+ok( $stderr =~ /index uniqueness is violated for index "bttest_unique_idx2"/,
+ 'detected uniqueness violation for index "bttest_unique_idx2"');
+
+#
+# Test 3.
+# - same as Test 2, but with index deduplication
+#
+# Then uniqueness violation is detected between different posting list
+# entries inside one index entry.
+#
+
+# Due to bad cmp function we expect amcheck to detect item order violation,
+# but no uniqueness violation.
+($result, $stdout, $stderr) = $node->psql(
+ 'postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok( $stderr =~ /item order invariant violated for index "bttest_unique_idx3"/,
+ 'detected item order invariant violation for index "bttest_unique_idx3"');
+
+# For unique index deduplication is possible only for same values, but
+# with different visibility.
+$node->safe_psql(
+ 'postgres', q(
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+ DELETE FROM bttest_unique3 WHERE 380 <= i AND i <= 420;
+ INSERT INTO bttest_unique3 (SELECT * FROM generate_series(380, 420));
+ INSERT INTO bttest_unique3 VALUES (400);
+));
+
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc SET
+ amproc = 'ok_cmp3'::regproc
+ WHERE amproc = 'bad_cmp3'::regproc;
+));
+
+($result, $stdout, $stderr) = $node->psql(
+ 'postgres', q(
+ SELECT bt_index_check('bttest_unique_idx3', true, true);
+));
+ok( $stderr =~ /index uniqueness is violated for index "bttest_unique_idx3"/,
+ 'detected uniqueness violation for index "bttest_unique_idx3"');
+
+$node->stop;
+done_testing();
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 3e07a3e35f4..7282cf7fc80 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -81,11 +81,19 @@ typedef struct BtreeCheckState
bool heapallindexed;
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
+ /* Also check uniqueness constraint if index is unique */
+ bool checkunique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
BufferAccessStrategy checkstrategy;
+ /*
+ * Info for uniqueness checking. Fill these fields once per index check.
+ */
+ IndexInfo *indexinfo;
+ Snapshot snapshot;
+
/*
* Mutable state, for verification of particular page:
*/
@@ -140,19 +148,33 @@ PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
- bool heapallindexed, bool rootdescend);
+ bool heapallindexed, bool rootdescend,
+ bool checkunique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend);
+ bool rootdescend, bool checkunique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
+static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
+static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
+ BlockNumber block, OffsetNumber offset,
+ int posting, ItemPointer nexttid,
+ BlockNumber nblock, OffsetNumber noffset,
+ int nposting);
+static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock,
+ OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid,
+ OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block);
static void bt_target_page_check(BtreeCheckState *state);
-static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state);
+static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
+ OffsetNumber *rightfirstoffset);
static void bt_child_check(BtreeCheckState *state, BTScanInsert targetkey,
OffsetNumber downlinkoffnum);
static void bt_child_highkey_check(BtreeCheckState *state,
@@ -192,7 +214,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -205,17 +227,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
+ bool checkunique = false;
- if (PG_NARGS() == 2)
+ if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
+ if (PG_NARGS() == 3)
+ checkunique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -229,13 +254,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
+ bool checkunique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
- if (PG_NARGS() == 3)
+ if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
+ if (PG_NARGS() == 4)
+ checkunique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
PG_RETURN_VOID();
}
@@ -245,7 +273,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend)
+ bool rootdescend, bool checkunique)
{
Oid heapid;
Relation indrel;
@@ -356,7 +384,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend);
+ heapallindexed, rootdescend, checkunique);
}
/* Roll back any GUC changes executed by index functions */
@@ -457,7 +485,8 @@ btree_index_mainfork_expected(Relation rel)
*/
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
- bool readonly, bool heapallindexed, bool rootdescend)
+ bool readonly, bool heapallindexed, bool rootdescend,
+ bool checkunique)
{
BtreeCheckState *state;
Page metapage;
@@ -489,6 +518,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
+ state->checkunique = checkunique;
+ state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
{
@@ -546,6 +577,23 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
}
+ /*
+ * We need a snapshot to check the uniqueness of the index. For better
+ * performance take it once per index check. If snapshot already taken
+ * reuse it.
+ */
+ if (state->checkunique)
+ {
+ state->indexinfo = BuildIndexInfo(state->rel);
+ if (state->indexinfo->ii_Unique)
+ {
+ if (snapshot != SnapshotAny)
+ state->snapshot = snapshot;
+ else
+ state->snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ }
+ }
+
Assert(!state->rootdescend || state->readonly);
if (state->rootdescend && !state->heapkeyspace)
ereport(ERROR,
@@ -672,6 +720,8 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
}
/* Be tidy: */
+ if (snapshot == SnapshotAny && state->snapshot != InvalidSnapshot)
+ UnregisterSnapshot(state->snapshot);
MemoryContextDelete(state->targetcontext);
}
@@ -912,6 +962,161 @@ nextpage:
return nextleveldown;
}
+/* Check visibility of the table entry referenced by nbtree index */
+static bool
+heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
+{
+ bool tid_visible;
+
+ TupleTableSlot *slot = table_slot_create(state->heaprel, NULL);
+
+ tid_visible = table_tuple_fetch_row_version(state->heaprel,
+ tid, state->snapshot, slot);
+ if (slot != NULL)
+ ExecDropSingleTupleTableSlot(slot);
+
+ return tid_visible;
+}
+
+/*
+ * Prepare an error message for unique constrain violation in
+ * a btree index and report ERROR.
+ */
+static void
+bt_report_duplicate(BtreeCheckState *state,
+ ItemPointer tid, BlockNumber block, OffsetNumber offset,
+ int posting,
+ ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
+ int nposting)
+{
+ char *htid,
+ *nhtid,
+ *itid,
+ *nitid = "",
+ *pposting = "",
+ *pnposting = "";
+
+ htid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(tid),
+ ItemPointerGetOffsetNumberNoCheck(tid));
+ nhtid = psprintf("tid=(%u,%u)",
+ ItemPointerGetBlockNumberNoCheck(nexttid),
+ ItemPointerGetOffsetNumberNoCheck(nexttid));
+ itid = psprintf("tid=(%u,%u)", block, offset);
+
+ if (nblock != block || noffset != offset)
+ nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
+
+ if (posting >= 0)
+ pposting = psprintf(" posting %u", posting);
+
+ if (nposting >= 0)
+ pnposting = psprintf(" posting %u", nposting);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("index uniqueness is violated for index \"%s\"",
+ RelationGetRelationName(state->rel)),
+ errdetail("Index %s%s and%s%s (point to heap %s and %s) page lsn=%X/%X.",
+ itid, pposting, nitid, pnposting, htid, nhtid,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+}
+
+/* Check if current nbtree leaf entry complies with UNIQUE constraint */
+static void
+bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
+ BlockNumber targetblock, OffsetNumber offset, int *lVis_i,
+ ItemPointer *lVis_tid, OffsetNumber *lVis_offset,
+ BlockNumber *lVis_block)
+{
+ ItemPointer tid;
+ bool has_visible_entry = false;
+
+ Assert(targetblock != P_NONE);
+
+ /*
+ * Current tuple has posting list. Report duplicate if TID of any posting
+ * list entry is visible and lVis_tid is valid.
+ */
+ if (BTreeTupleIsPosting(itup))
+ {
+ for (int i = 0; i < BTreeTupleGetNPosting(itup); i++)
+ {
+ tid = BTreeTupleGetPostingN(itup, i);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, i);
+ }
+
+ /*
+ * Prevent double reporting unique constraint violation between
+ * the posting list entries of the first tuple on the page after
+ * cross-page check.
+ */
+ if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ return;
+
+ *lVis_i = i;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+ }
+
+ /*
+ * Current tuple has no posting list. If TID is visible save info about
+ * it for the next comparisons in the loop in bt_page_check(). Report
+ * duplicate if lVis_tid is already valid.
+ */
+ else
+ {
+ tid = BTreeTupleGetHeapTID(itup);
+ if (heap_entry_is_visible(state, tid))
+ {
+ has_visible_entry = true;
+ if (ItemPointerIsValid(*lVis_tid))
+ {
+ bt_report_duplicate(state,
+ *lVis_tid, *lVis_block,
+ *lVis_offset, *lVis_i,
+ tid, targetblock,
+ offset, -1);
+ }
+ *lVis_i = -1;
+ *lVis_tid = tid;
+ *lVis_offset = offset;
+ *lVis_block = targetblock;
+ }
+ }
+
+ if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
+ *lVis_block != targetblock)
+ {
+ char *posting = "";
+
+ if (*lVis_i >= 0)
+ posting = psprintf(" posting %u", *lVis_i);
+ ereport(DEBUG1,
+ (errcode(ERRCODE_NO_DATA),
+ errmsg("index uniqueness can not be checked for index tid=(%u,%u) in index \"%s\"",
+ targetblock, offset,
+ RelationGetRelationName(state->rel)),
+ errdetail("It doesn't have visible heap tids and key is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)).",
+ *lVis_block, *lVis_offset, posting,
+ ItemPointerGetBlockNumberNoCheck(*lVis_tid),
+ ItemPointerGetOffsetNumberNoCheck(*lVis_tid)),
+ errhint("VACUUM the table and repeat the check.")));
+ }
+}
+
/*
* Raise an error when target page's left link does not point back to the
* previous target page, called leftcurrent here. The leftcurrent page's
@@ -1066,6 +1271,9 @@ bt_recheck_sibling_links(BtreeCheckState *state,
* - Various checks on the structure of tuples themselves. For example, check
* that non-pivot tuples have no truncated attributes.
*
+ * - For index with unique constraint make sure that only one of table entries
+ * for equal keys is visible.
+ *
* Furthermore, when state passed shows ShareLock held, function also checks:
*
* - That all child pages respect strict lower bound from parent's pivot
@@ -1088,6 +1296,13 @@ bt_target_page_check(BtreeCheckState *state)
OffsetNumber max;
BTPageOpaque topaque;
+ /* last visible entry info for checking indexes with unique constraint */
+ int lVis_i = -1; /* the position of last visible item for
+ * posting tuple. for non-posting tuple (-1) */
+ ItemPointer lVis_tid = NULL;
+ BlockNumber lVis_block = InvalidBlockNumber;
+ OffsetNumber lVis_offset = InvalidOffsetNumber;
+
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1478,6 +1693,45 @@ bt_target_page_check(BtreeCheckState *state)
LSN_FORMAT_ARGS(state->targetlsn))));
}
+ /*
+ * If the index is unique verify entries uniqueness by checking the heap
+ * tuples visibility.
+ */
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ P_ISLEAF(topaque) && !skey->anynullkeys)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ P_ISLEAF(topaque) && OffsetNumberNext(offset) <= max)
+ {
+ /* Save current scankey tid */
+ scantid = skey->scantid;
+
+ /*
+ * Invalidate scankey tid to make _bt_compare compare only keys in
+ * the item to report equality even if heap TIDs are different
+ */
+ skey->scantid = NULL;
+
+ /*
+ * If next key tuple is different, invalidate last visible entry
+ * data (whole index tuple or last posting in index tuple). Key
+ * containing null value does not violate unique constraint and
+ * treated as different to any other key.
+ */
+ if (_bt_compare(state->rel, skey, state->target,
+ OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
+ {
+ lVis_i = -1;
+ lVis_tid = NULL;
+ lVis_block = InvalidBlockNumber;
+ lVis_offset = InvalidOffsetNumber;
+ }
+ skey->scantid = scantid; /* Restore saved scan key state */
+ }
+
/*
* * Last item check *
*
@@ -1495,12 +1749,16 @@ bt_target_page_check(BtreeCheckState *state)
* available from sibling for various reasons, though (e.g., target is
* the rightmost page on level).
*/
- else if (offset == max)
+ if (offset == max)
{
BTScanInsert rightkey;
+ BlockNumber rightblock_number;
+
+ /* first offset on a right index page (log only) */
+ OffsetNumber rightfirstoffset = InvalidOffsetNumber;
/* Get item in next/right page */
- rightkey = bt_right_page_check_scankey(state);
+ rightkey = bt_right_page_check_scankey(state, &rightfirstoffset);
if (rightkey &&
!invariant_g_offset(state, rightkey, max))
@@ -1534,6 +1792,45 @@ bt_target_page_check(BtreeCheckState *state)
state->targetblock, offset,
LSN_FORMAT_ARGS(state->targetlsn))));
}
+
+ /*
+ * If index has unique constraint make sure that no more than one
+ * found equal items is visible.
+ */
+ rightblock_number = topaque->btpo_next;
+ if (state->checkunique && state->indexinfo->ii_Unique &&
+ rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ {
+ elog(DEBUG2, "check cross page unique condition");
+
+ /*
+ * Make _bt_compare compare only index keys without heap TIDs.
+ * rightkey->scantid is modified destructively but it is ok
+ * for it is not used later.
+ */
+ rightkey->scantid = NULL;
+
+ /* The first key on the next page is the same */
+ if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
+ {
+ elog(DEBUG2, "cross page equal keys");
+ state->target = palloc_btree_page(state,
+ rightblock_number);
+ topaque = BTPageGetOpaque(state->target);
+
+ if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
+ break;
+
+ itemid = PageGetItemIdCareful(state, rightblock_number,
+ state->target,
+ rightfirstoffset);
+ itup = (IndexTuple) PageGetItem(state->target, itemid);
+
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
+ }
}
/*
@@ -1579,9 +1876,11 @@ bt_target_page_check(BtreeCheckState *state)
*
* Note that !readonly callers must reverify that target page has not
* been concurrently deleted.
+ *
+ * Save rightfirstdataoffset for detailed error message.
*/
static BTScanInsert
-bt_right_page_check_scankey(BtreeCheckState *state)
+bt_right_page_check_scankey(BtreeCheckState *state, OffsetNumber *rightfirstoffset)
{
BTPageOpaque opaque;
ItemId rightitem;
@@ -1748,6 +2047,7 @@ bt_right_page_check_scankey(BtreeCheckState *state)
/* Return first data item (if any) */
rightitem = PageGetItemIdCareful(state, targetnext, rightpage,
P_FIRSTDATAKEY(opaque));
+ *rightfirstoffset = P_FIRSTDATAKEY(opaque);
}
else if (!P_ISLEAF(opaque) &&
nline >= OffsetNumberNext(P_FIRSTDATAKEY(opaque)))
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 2b9c1a9205f..780fd05a73b 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -58,7 +58,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -115,7 +115,10 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When a routine, lightweight test for
+ <literal>true</literal>. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When a routine, lightweight test for
corruption is required in a live production environment, using
<function>bt_index_check</function> often provides the best
trade-off between thoroughness of verification and limiting the
@@ -126,7 +129,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -139,7 +142,10 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When the optional <parameter>rootdescend</parameter>
+ index. When <parameter>checkunique</parameter>
+ is <literal>true</literal> <function>bt_index_parent_check</function> will
+ check that no more than one among duplicate entries in unique
+ index is visible. When the optional <parameter>rootdescend</parameter>
argument is <literal>true</literal>, verification re-finds
tuples on the leaf level by performing a new search from the
root page for each tuple. The checks that can be performed by
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index 20c2897accb..067c806b46d 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -432,6 +432,17 @@ PostgreSQL documentation
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><option>--checkunique</option></term>
+ <listitem>
+ <para>
+ For each index with unique constraint checked, verify that no more than
+ one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
+ <option>checkunique</option> option.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 8ac7051ff4d..57c7c1917c4 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,6 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
+ bool checkunique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -132,6 +133,7 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
+ .checkunique = false,
.no_btree_expansion = false
};
@@ -148,6 +150,7 @@ typedef struct DatabaseInfo
{
char *datname;
char *amcheck_schema; /* escaped, quoted literal */
+ bool is_checkunique;
} DatabaseInfo;
typedef struct RelationInfo
@@ -267,6 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
+ {"checkunique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -434,6 +438,9 @@ main(int argc, char *argv[])
if (optarg)
opts.install_schema = pg_strdup(optarg);
break;
+ case 14:
+ opts.checkunique = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -589,6 +596,38 @@ main(int argc, char *argv[])
PQdb(conn), PQgetvalue(result, 0, 1), amcheck_schema);
dat->amcheck_schema = PQescapeIdentifier(conn, amcheck_schema,
strlen(amcheck_schema));
+
+ /*
+ * Check the version of amcheck extension. Skip requested unique
+ * constraint check with warning if it is not yet supported by amcheck.
+ */
+ if (opts.checkunique == true)
+ {
+ /*
+ * Now amcheck has only major and minor versions in the string but
+ * we also support revision just in case. Now it is expected to be
+ * zero.
+ */
+ int vmaj = 0,
+ vmin = 0,
+ vrev = 0;
+ const char *amcheck_version = PQgetvalue(result, 0, 1);
+
+ sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
+
+ /*
+ * checkunique option is supported in amcheck since version 1.4
+ */
+ if ((vmaj == 1 && vmin < 4) || vmaj == 0)
+ {
+ pg_log_warning("--checkunique option is not supported by amcheck "
+ "version \"%s\"", amcheck_version);
+ dat->is_checkunique = false;
+ }
+ else
+ dat->is_checkunique = true;
+ }
+
PQclear(result);
compile_relation_list_one_db(conn, &relations, dat, &pagestotal);
@@ -845,7 +884,8 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
if (opts.parent_check)
appendPQExpBuffer(sql,
"SELECT %s.bt_index_parent_check("
- "index := c.oid, heapallindexed := %s, rootdescend := %s)"
+ "index := c.oid, heapallindexed := %s, rootdescend := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -854,11 +894,13 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
"SELECT %s.bt_index_check("
- "index := c.oid, heapallindexed := %s)"
+ "index := c.oid, heapallindexed := %s "
+ "%s)"
"\nFROM pg_catalog.pg_class c, pg_catalog.pg_index i "
"WHERE c.oid = %u "
"AND c.oid = i.indexrelid "
@@ -866,6 +908,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
+ (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
rel->reloid);
}
@@ -1163,6 +1206,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
+ printf(_(" --checkunique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index d577cffa30d..2b7ef198552 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -257,6 +257,9 @@ for my $dbname (qw(db1 db2 db3))
CREATE INDEX t1_spgist ON $schema.t1 USING SPGIST (ir);
CREATE INDEX t2_spgist ON $schema.t2 USING SPGIST (ir);
+
+ CREATE UNIQUE INDEX t1_btree_unique ON $schema.t1 USING BTREE (i);
+ CREATE UNIQUE INDEX t2_btree_unique ON $schema.t2 USING BTREE (i);
));
}
}
@@ -517,4 +520,51 @@ $node->command_checks_all(
0, [$no_output_re], [$no_output_re],
'pg_amcheck excluding all corrupt schemas');
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
+ '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --parent-check --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
+ '--rootdescend', '--checkunique', 'db1'
+ ],
+ 2,
+ [$index_missing_relation_fork_re],
+ [$no_output_re],
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+
+$node->command_checks_all(
+ [
+ @cmd, '--checkunique', '-d', 'db1', '-d', 'db2',
+ '-d', 'db3', '-S', 's*'
+ ],
+ 0,
+ [$no_output_re],
+ [$no_output_re],
+ 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+
+#
+# Smoke test for checkunique option for not supported versions.
+#
+$node->safe_psql(
+ 'db3', q(
+ DROP EXTENSION amcheck;
+ CREATE EXTENSION amcheck WITH SCHEMA amcheck_schema VERSION '1.3' ;
+));
+
+$node->command_checks_all(
+ [ @cmd, '--checkunique', 'db3' ],
+ 0,
+ [$no_output_re],
+ [
+ qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ ],
+ 'pg_amcheck smoke test --checkunique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index fd476179f49..a5ef2c0f33d 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -22,14 +22,33 @@ $node->safe_psql(
CREATE FUNCTION int4_asc_cmp (a int4, b int4) RETURNS int LANGUAGE sql AS $$
SELECT CASE WHEN $1 = $2 THEN 0 WHEN $1 > $2 THEN 1 ELSE -1 END; $$;
+ CREATE FUNCTION ok_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
CREATE OPERATOR CLASS int4_fickle_ops FOR TYPE int4 USING btree AS
OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
OPERATOR 5 > (int4, int4), FUNCTION 1 int4_asc_cmp(int4, int4);
+ CREATE OPERATOR CLASS int4_unique_ops FOR TYPE int4 USING btree AS
+ OPERATOR 1 < (int4, int4), OPERATOR 2 <= (int4, int4),
+ OPERATOR 3 = (int4, int4), OPERATOR 4 >= (int4, int4),
+ OPERATOR 5 > (int4, int4), FUNCTION 1 ok_cmp(int4, int4);
+
CREATE TABLE int4tbl (i int4);
INSERT INTO int4tbl (SELECT * FROM generate_series(1,1000) gs);
CREATE INDEX fickleidx ON int4tbl USING btree (i int4_fickle_ops);
+ CREATE UNIQUE INDEX bttest_unique_idx
+ ON int4tbl
+ USING btree (i int4_unique_ops)
+ WITH (deduplicate_items = off);
));
# We have not yet broken the index, so we should get no corruption
@@ -57,4 +76,50 @@ $node->command_checks_all(
'pg_amcheck all schemas, tables and indexes reports fickleidx corruption'
);
+#
+# Check unique constraints
+#
+
+# Repair broken opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'int4_asc_cmp'::regproc
+ WHERE amproc = 'int4_desc_cmp'::regproc
+));
+
+# We should get no corruptions
+$node->command_like(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ qr/^$/,
+ 'pg_amcheck all schemas, tables and indexes reports no corruption');
+
+# Break opclass for check unique tests.
+$node->safe_psql(
+ 'postgres', q(
+ CREATE FUNCTION bad_cmp (int4, int4)
+ RETURNS int LANGUAGE sql AS
+ $$
+ SELECT
+ CASE WHEN ($1 = 768 AND $2 = 769) OR
+ ($1 = 769 AND $2 = 768) THEN 0
+ WHEN $1 < $2 THEN -1
+ WHEN $1 > $2 THEN 1
+ ELSE 0
+ END;
+ $$;
+
+ UPDATE pg_catalog.pg_amproc
+ SET amproc = 'bad_cmp'::regproc
+ WHERE amproc = 'ok_cmp'::regproc
+));
+
+# Unique index corruption should now be reported
+$node->command_checks_all(
+ [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ 2,
+ [qr/index uniqueness is violated for index "bttest_unique_idx"/],
+ [],
+ 'pg_amcheck all schemas, tables and indexes reports bttest_unique_idx corruption'
+);
done_testing();
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
@ 2023-10-30 07:29 ` Pavel Borisov <[email protected]>
1 sibling, 0 replies; 45+ messages in thread
From: Pavel Borisov @ 2023-10-30 07:29 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Hamid Akhtar <[email protected]>
Hi, Alexander!
On Wed, 25 Oct 2023 at 00:13, Alexander Korotkov <[email protected]> wrote:
>
> On Wed, Sep 28, 2022 at 11:44 AM Aleksander Alekseev
> <[email protected]> wrote:
> > > I think, this patch was marked as "Waiting on Author", probably, by mistake. Since recent changes were done without any significant code changes and CF bot how happy again.
> > >
> > > I'm going to move it to RfC, could I? If not, please tell why.
> >
> > I restored the "Ready for Committer" state. I don't think it's a good
> > practice to change the state every time the patch has a slight
> > conflict or something. This is not helpful at all. Such things happen
> > quite regularly and typically are fixed in a couple of days.
>
> This patch seems useful to me. I went through the thread, it seems
> that all the critics are addressed.
>
> I've rebased this patch. Also, I've run perltidy for tests, split
> long errmsg() into errmsg(), errdetail() and errhint(), and do other
> minor enchantments.
>
> I think this patch is ready to go. I'm going to push it if there are
> no objections.
>
> ------
> Regards,
> Alexander Korotkov
It's very good that this long-standing patch is finally committed. Thanks a lot!
Regards,
Pavel Borisov
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
@ 2024-04-17 06:38 ` Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Peter Eisentraut @ 2024-04-17 06:38 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; Aleksander Alekseev <[email protected]>; +Cc: Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Pavel Borisov <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]; Hamid Akhtar <[email protected]>
On 24.10.23 22:13, Alexander Korotkov wrote:
> On Wed, Sep 28, 2022 at 11:44 AM Aleksander Alekseev
> <[email protected]> wrote:
>>> I think, this patch was marked as "Waiting on Author", probably, by mistake. Since recent changes were done without any significant code changes and CF bot how happy again.
>>>
>>> I'm going to move it to RfC, could I? If not, please tell why.
>>
>> I restored the "Ready for Committer" state. I don't think it's a good
>> practice to change the state every time the patch has a slight
>> conflict or something. This is not helpful at all. Such things happen
>> quite regularly and typically are fixed in a couple of days.
>
> This patch seems useful to me. I went through the thread, it seems
> that all the critics are addressed.
>
> I've rebased this patch. Also, I've run perltidy for tests, split
> long errmsg() into errmsg(), errdetail() and errhint(), and do other
> minor enchantments.
>
> I think this patch is ready to go. I'm going to push it if there are
> no objections.
I just found the new pg_amcheck option --checkunique in PG17-to-be.
Could we rename this to --check-unique? Seems friendlier. Maybe also
rename the bt_index_check function argument to check_unique.
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
@ 2024-04-24 09:58 ` Alexander Korotkov <[email protected]>
2024-04-25 12:59 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Alexander Korotkov @ 2024-04-24 09:58 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Pavel Borisov <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
On Wed, Apr 17, 2024 at 9:38 AM Peter Eisentraut <[email protected]> wrote:
> On 24.10.23 22:13, Alexander Korotkov wrote:
> > On Wed, Sep 28, 2022 at 11:44 AM Aleksander Alekseev
> > <[email protected]> wrote:
> >>> I think, this patch was marked as "Waiting on Author", probably, by mistake. Since recent changes were done without any significant code changes and CF bot how happy again.
> >>>
> >>> I'm going to move it to RfC, could I? If not, please tell why.
> >>
> >> I restored the "Ready for Committer" state. I don't think it's a good
> >> practice to change the state every time the patch has a slight
> >> conflict or something. This is not helpful at all. Such things happen
> >> quite regularly and typically are fixed in a couple of days.
> >
> > This patch seems useful to me. I went through the thread, it seems
> > that all the critics are addressed.
> >
> > I've rebased this patch. Also, I've run perltidy for tests, split
> > long errmsg() into errmsg(), errdetail() and errhint(), and do other
> > minor enchantments.
> >
> > I think this patch is ready to go. I'm going to push it if there are
> > no objections.
>
> I just found the new pg_amcheck option --checkunique in PG17-to-be.
> Could we rename this to --check-unique? Seems friendlier. Maybe also
> rename the bt_index_check function argument to check_unique.
+1 from me
Let's do so if nobody objects.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
@ 2024-04-25 12:59 ` Pavel Borisov <[email protected]>
2024-04-25 13:44 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Karina Litskevich <[email protected]>
2024-05-01 02:24 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Noah Misch <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Pavel Borisov @ 2024-04-25 12:59 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
Hi, hackers!
On Wed, 24 Apr 2024 at 13:58, Alexander Korotkov <[email protected]>
wrote:
> On Wed, Apr 17, 2024 at 9:38 AM Peter Eisentraut <[email protected]>
> wrote:
> > On 24.10.23 22:13, Alexander Korotkov wrote:
> > > On Wed, Sep 28, 2022 at 11:44 AM Aleksander Alekseev
> > > <[email protected]> wrote:
> > >>> I think, this patch was marked as "Waiting on Author", probably, by
> mistake. Since recent changes were done without any significant code
> changes and CF bot how happy again.
> > >>>
> > >>> I'm going to move it to RfC, could I? If not, please tell why.
> > >>
> > >> I restored the "Ready for Committer" state. I don't think it's a good
> > >> practice to change the state every time the patch has a slight
> > >> conflict or something. This is not helpful at all. Such things happen
> > >> quite regularly and typically are fixed in a couple of days.
> > >
> > > This patch seems useful to me. I went through the thread, it seems
> > > that all the critics are addressed.
> > >
> > > I've rebased this patch. Also, I've run perltidy for tests, split
> > > long errmsg() into errmsg(), errdetail() and errhint(), and do other
> > > minor enchantments.
> > >
> > > I think this patch is ready to go. I'm going to push it if there are
> > > no objections.
> >
> > I just found the new pg_amcheck option --checkunique in PG17-to-be.
> > Could we rename this to --check-unique? Seems friendlier. Maybe also
> > rename the bt_index_check function argument to check_unique.
>
> +1 from me
> Let's do so if nobody objects.
>
Thank you very much for your input in this thread!
See the patches based on the proposals in the attachment:
0001: Optimize speed by avoiding heap visibility checking for different
non-deduplicated index tuples as proposed by Noah Misch
Speed measurements on my laptop using the exact method recommended by Noah
upthread:
Current master branch: checkunique off: 144s, checkunique on: 419s
With patch 0001: checkunique off: 141s, checkunique on: 171s
0002: Use structure to store and transfer info about last visible heap
entry (code refactoring) as proposed by Alexander Korotkov
0003: Don't load rightpage into BtreeCheckState (code refactoring) as
proposed by Peter Geoghegan
Loading of right page for cross-page unique constraint check in the same
way as in bt_right_page_check_scankey()
0004: Report error when next page to a leaf is not a leaf as proposed by
Peter Geoghegan
I think it's a very improbable condition and this check might be not
necessary, but it's right and safe to break check and report error.
0005: Rename checkunique parameter to more user friendly as proposed by
Peter Eisentraut and Alexander Korotkov
Again many thanks for the useful proposals!
Regards,
Pavel Borisov,
Supabase
Attachments:
[application/octet-stream] v1-0004-Amcheck-Report-error-when-next-page-to-a-leaf-is-.patch (2.5K, ../../CALT9ZEFY-vsL4-cxta49wP=H7RjDaZk+Wg-ft4m-kVS_=WMsAQ@mail.gmail.com/3-v1-0004-Amcheck-Report-error-when-next-page-to-a-leaf-is-.patch)
download | inline diff:
From a2950357675198285dfdbc974346bce12dbd3c55 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Thu, 25 Apr 2024 15:06:36 +0400
Subject: [PATCH v1 4/5] Amcheck: Report error when next page to a leaf is not
a leaf
This is a very unlikely condition during checking unique constraint,
meaning that index connectivity is violated badly and we shouldn't
continue checking to avoid neverending loops etc. So it's better
to honestly throw an error.
Reported-by: Peter Geoghegan
---
contrib/amcheck/verify_nbtree.c | 23 ++++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index d6f70206db..dfc9ed769f 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -1829,7 +1829,6 @@ bt_target_page_check(BtreeCheckState *state)
if (offset == max)
{
BTScanInsert rightkey;
- BlockNumber rightblock_number;
/* first offset on a right index page (log only) */
OffsetNumber rightfirstoffset = InvalidOffsetNumber;
@@ -1874,12 +1873,12 @@ bt_target_page_check(BtreeCheckState *state)
* If index has unique constraint make sure that no more than one
* found equal items is visible.
*/
- rightblock_number = topaque->btpo_next;
if (state->checkunique && state->indexinfo->ii_Unique &&
- rightkey && P_ISLEAF(topaque) && rightblock_number != P_NONE)
+ rightkey && P_ISLEAF(topaque) && !P_RIGHTMOST(topaque))
{
- elog(DEBUG2, "check cross page unique condition");
+ BlockNumber rightblock_number = topaque->btpo_next;
+ elog(DEBUG2, "check cross page unique condition");
/*
* Make _bt_compare compare only index keys without heap TIDs.
* rightkey->scantid is modified destructively but it is ok
@@ -1900,9 +1899,19 @@ bt_target_page_check(BtreeCheckState *state)
rightblock_number);
topaque = BTPageGetOpaque(rightpage);
- if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
- break;
-
+ if (P_IGNORE(topaque))
+ {
+ if (unlikely(!P_ISLEAF(topaque)))
+ ereport(ERROR,
+ (errcode(ERRCODE_INDEX_CORRUPTED),
+ errmsg("right block of leaf block is non-leaf for index \"%s\"",
+ RelationGetRelationName(state->rel)),
+ errdetail_internal("Block=%u page lsn=%X/%X.",
+ state->targetblock,
+ LSN_FORMAT_ARGS(state->targetlsn))));
+ else
+ break;
+ }
itemid = PageGetItemIdCareful(state, rightblock_number,
rightpage,
rightfirstoffset);
--
2.34.1
[application/octet-stream] v1-0002-Amcheck-code-refactoring.patch (9.7K, ../../CALT9ZEFY-vsL4-cxta49wP=H7RjDaZk+Wg-ft4m-kVS_=WMsAQ@mail.gmail.com/4-v1-0002-Amcheck-code-refactoring.patch)
download | inline diff:
From 92a085925e84cd6c34c59861f2775e53a1048665 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Thu, 25 Apr 2024 14:07:45 +0400
Subject: [PATCH v1 2/5] Amcheck: code refactoring
Use structure to store and transfer info about last visible heap entry
among the equal index/posting list entries.
Reported-by: Alexander Korotkov
Discussion: https://www.postgresql.org/message-id/CAPpHfdsVbB9ToriaB1UHuOKwjKxiZmTFQcEF%3DjuzzC_nby31uA%40mail.gmail.com
---
contrib/amcheck/verify_nbtree.c | 112 ++++++++++++++------------------
1 file changed, 49 insertions(+), 63 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index ae8012f15b..66f2e619a8 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -145,6 +145,15 @@ typedef struct BtreeLevel
bool istruerootlevel;
} BtreeLevel;
+/* Info for last visible entry for checking unique constraint */
+typedef struct lVisInfo
+{
+ ItemPointer tid; /* Heap tid */
+ BlockNumber block; /* Index block */
+ OffsetNumber offset; /* Offset on index block */
+ int i; /* Number in posting list. (-1 for non-deduplicated) */
+} lVisInfo;
+
PG_FUNCTION_INFO_V1(bt_index_check);
PG_FUNCTION_INFO_V1(bt_index_parent_check);
@@ -165,17 +174,13 @@ static void bt_recheck_sibling_links(BtreeCheckState *state,
BlockNumber btpo_prev_from_target,
BlockNumber leftcurrent);
static bool heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid);
-static void bt_report_duplicate(BtreeCheckState *state, ItemPointer tid,
- BlockNumber block, OffsetNumber offset,
- int posting, ItemPointer nexttid,
+static void bt_report_duplicate(BtreeCheckState *state, lVisInfo *lVis,
+ ItemPointer nexttid,
BlockNumber nblock, OffsetNumber noffset,
int nposting);
static void bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
BlockNumber targetblock,
- OffsetNumber offset, int *lVis_i,
- ItemPointer *lVis_tid,
- OffsetNumber *lVis_offset,
- BlockNumber *lVis_block);
+ OffsetNumber offset, lVisInfo *lVis);
static void bt_target_page_check(BtreeCheckState *state);
static BTScanInsert bt_right_page_check_scankey(BtreeCheckState *state,
OffsetNumber *rightfirstoffset);
@@ -997,8 +1002,7 @@ heap_entry_is_visible(BtreeCheckState *state, ItemPointer tid)
*/
static void
bt_report_duplicate(BtreeCheckState *state,
- ItemPointer tid, BlockNumber block, OffsetNumber offset,
- int posting,
+ lVisInfo *lVis,
ItemPointer nexttid, BlockNumber nblock, OffsetNumber noffset,
int nposting)
{
@@ -1010,18 +1014,18 @@ bt_report_duplicate(BtreeCheckState *state,
*pnposting = "";
htid = psprintf("tid=(%u,%u)",
- ItemPointerGetBlockNumberNoCheck(tid),
- ItemPointerGetOffsetNumberNoCheck(tid));
+ ItemPointerGetBlockNumberNoCheck(lVis->tid),
+ ItemPointerGetOffsetNumberNoCheck(lVis->tid));
nhtid = psprintf("tid=(%u,%u)",
ItemPointerGetBlockNumberNoCheck(nexttid),
ItemPointerGetOffsetNumberNoCheck(nexttid));
- itid = psprintf("tid=(%u,%u)", block, offset);
+ itid = psprintf("tid=(%u,%u)", lVis->block, lVis->offset);
- if (nblock != block || noffset != offset)
+ if (nblock != lVis->block || noffset != lVis->offset)
nitid = psprintf(" tid=(%u,%u)", nblock, noffset);
- if (posting >= 0)
- pposting = psprintf(" posting %u", posting);
+ if (lVis->i >= 0)
+ pposting = psprintf(" posting %u", lVis->i);
if (nposting >= 0)
pnposting = psprintf(" posting %u", nposting);
@@ -1038,9 +1042,7 @@ bt_report_duplicate(BtreeCheckState *state,
/* Check if current nbtree leaf entry complies with UNIQUE constraint */
static void
bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
- BlockNumber targetblock, OffsetNumber offset, int *lVis_i,
- ItemPointer *lVis_tid, OffsetNumber *lVis_offset,
- BlockNumber *lVis_block)
+ BlockNumber targetblock, OffsetNumber offset, lVisInfo *lVis)
{
ItemPointer tid;
bool has_visible_entry = false;
@@ -1049,7 +1051,7 @@ bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
/*
* Current tuple has posting list. Report duplicate if TID of any posting
- * list entry is visible and lVis_tid is valid.
+ * list entry is visible and lVis->tid is valid.
*/
if (BTreeTupleIsPosting(itup))
{
@@ -1059,11 +1061,10 @@ bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
if (heap_entry_is_visible(state, tid))
{
has_visible_entry = true;
- if (ItemPointerIsValid(*lVis_tid))
+ if (ItemPointerIsValid(lVis->tid))
{
bt_report_duplicate(state,
- *lVis_tid, *lVis_block,
- *lVis_offset, *lVis_i,
+ lVis,
tid, targetblock,
offset, i);
}
@@ -1073,13 +1074,13 @@ bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
* between the posting list entries of the first tuple on the
* page after cross-page check.
*/
- if (*lVis_block != targetblock && ItemPointerIsValid(*lVis_tid))
+ if (lVis->block != targetblock && ItemPointerIsValid(lVis->tid))
return;
- *lVis_i = i;
- *lVis_tid = tid;
- *lVis_offset = offset;
- *lVis_block = targetblock;
+ lVis->i = i;
+ lVis->tid = tid;
+ lVis->offset = offset;
+ lVis->block = targetblock;
}
}
}
@@ -1095,37 +1096,36 @@ bt_entry_unique_check(BtreeCheckState *state, IndexTuple itup,
if (heap_entry_is_visible(state, tid))
{
has_visible_entry = true;
- if (ItemPointerIsValid(*lVis_tid))
+ if (ItemPointerIsValid(lVis->tid))
{
bt_report_duplicate(state,
- *lVis_tid, *lVis_block,
- *lVis_offset, *lVis_i,
+ lVis,
tid, targetblock,
offset, -1);
}
- *lVis_i = -1;
- *lVis_tid = tid;
- *lVis_offset = offset;
- *lVis_block = targetblock;
+ lVis->i = -1;
+ lVis->tid = tid;
+ lVis->offset = offset;
+ lVis->block = targetblock;
}
}
- if (!has_visible_entry && *lVis_block != InvalidBlockNumber &&
- *lVis_block != targetblock)
+ if (!has_visible_entry && lVis->block != InvalidBlockNumber &&
+ lVis->block != targetblock)
{
char *posting = "";
- if (*lVis_i >= 0)
- posting = psprintf(" posting %u", *lVis_i);
+ if (lVis->i >= 0)
+ posting = psprintf(" posting %u", lVis->i);
ereport(DEBUG1,
(errcode(ERRCODE_NO_DATA),
errmsg("index uniqueness can not be checked for index tid=(%u,%u) in index \"%s\"",
targetblock, offset,
RelationGetRelationName(state->rel)),
errdetail("It doesn't have visible heap tids and key is equal to the tid=(%u,%u)%s (points to heap tid=(%u,%u)).",
- *lVis_block, *lVis_offset, posting,
- ItemPointerGetBlockNumberNoCheck(*lVis_tid),
- ItemPointerGetOffsetNumberNoCheck(*lVis_tid)),
+ lVis->block, lVis->offset, posting,
+ ItemPointerGetBlockNumberNoCheck(lVis->tid),
+ ItemPointerGetOffsetNumberNoCheck(lVis->tid)),
errhint("VACUUM the table and repeat the check.")));
}
}
@@ -1373,11 +1373,7 @@ bt_target_page_check(BtreeCheckState *state)
BTPageOpaque topaque;
/* last visible entry info for checking indexes with unique constraint */
- int lVis_i = -1; /* the position of last visible item for
- * posting tuple. for non-posting tuple (-1) */
- ItemPointer lVis_tid = NULL;
- BlockNumber lVis_block = InvalidBlockNumber;
- OffsetNumber lVis_offset = InvalidOffsetNumber;
+ lVisInfo lVis = {NULL, InvalidBlockNumber, InvalidOffsetNumber, -1};
topaque = BTPageGetOpaque(state->target);
max = PageGetMaxOffsetNumber(state->target);
@@ -1776,11 +1772,9 @@ bt_target_page_check(BtreeCheckState *state)
*/
if (state->checkunique && state->indexinfo->ii_Unique &&
P_ISLEAF(topaque) && !skey->anynullkeys &&
- (BTreeTupleIsPosting(itup) || ItemPointerIsValid(lVis_tid)))
+ (BTreeTupleIsPosting(itup) || ItemPointerIsValid(lVis.tid)))
{
- bt_entry_unique_check(state, itup, state->targetblock, offset,
- &lVis_i, &lVis_tid, &lVis_offset,
- &lVis_block);
+ bt_entry_unique_check(state, itup, state->targetblock, offset, &lVis);
unique_checked = true;
}
@@ -1805,17 +1799,13 @@ bt_target_page_check(BtreeCheckState *state)
if (_bt_compare(state->rel, skey, state->target,
OffsetNumberNext(offset)) != 0 || skey->anynullkeys)
{
- lVis_i = -1;
- lVis_tid = NULL;
- lVis_block = InvalidBlockNumber;
- lVis_offset = InvalidOffsetNumber;
+ lVis = (lVisInfo) {NULL, InvalidBlockNumber, InvalidOffsetNumber, -1};
}
else if (!unique_checked)
{
- bt_entry_unique_check(state, itup, state->targetblock, offset,
- &lVis_i, &lVis_tid, &lVis_offset,
- &lVis_block);
+ bt_entry_unique_check(state, itup, state->targetblock, offset, &lVis);
}
+
skey->scantid = scantid; /* Restore saved scan key state */
}
@@ -1901,9 +1891,7 @@ bt_target_page_check(BtreeCheckState *state)
if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
{
if (!unique_checked)
- bt_entry_unique_check(state, itup, state->targetblock, offset,
- &lVis_i, &lVis_tid, &lVis_offset,
- &lVis_block);
+ bt_entry_unique_check(state, itup, state->targetblock, offset, &lVis);
elog(DEBUG2, "cross page equal keys");
state->target = palloc_btree_page(state,
@@ -1918,9 +1906,7 @@ bt_target_page_check(BtreeCheckState *state)
rightfirstoffset);
itup = (IndexTuple) PageGetItem(state->target, itemid);
- bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset,
- &lVis_i, &lVis_tid, &lVis_offset,
- &lVis_block);
+ bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset, &lVis);
}
}
}
--
2.34.1
[application/octet-stream] v1-0003-Amcheck-Don-t-load-rightpage-into-BtreeCheckState.patch (1.7K, ../../CALT9ZEFY-vsL4-cxta49wP=H7RjDaZk+Wg-ft4m-kVS_=WMsAQ@mail.gmail.com/5-v1-0003-Amcheck-Don-t-load-rightpage-into-BtreeCheckState.patch)
download | inline diff:
From 526409e76419b387d9d4a40e94881231f256fcc8 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Thu, 25 Apr 2024 14:20:45 +0400
Subject: [PATCH v1 3/5] Amcheck: Don't load rightpage into BtreeCheckState
For cross-page unique constraint check use a local variable in the
similar way as implemented in bt_right_page_check_scankey().
Reported-by: Peter Geoghegan
---
contrib/amcheck/verify_nbtree.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 66f2e619a8..d6f70206db 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -1890,23 +1890,27 @@ bt_target_page_check(BtreeCheckState *state)
/* The first key on the next page is the same */
if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
{
+ Page rightpage;
+
if (!unique_checked)
bt_entry_unique_check(state, itup, state->targetblock, offset, &lVis);
elog(DEBUG2, "cross page equal keys");
- state->target = palloc_btree_page(state,
+ rightpage = palloc_btree_page(state,
rightblock_number);
- topaque = BTPageGetOpaque(state->target);
+ topaque = BTPageGetOpaque(rightpage);
if (P_IGNORE(topaque) || !P_ISLEAF(topaque))
break;
itemid = PageGetItemIdCareful(state, rightblock_number,
- state->target,
+ rightpage,
rightfirstoffset);
- itup = (IndexTuple) PageGetItem(state->target, itemid);
+ itup = (IndexTuple) PageGetItem(rightpage, itemid);
bt_entry_unique_check(state, itup, rightblock_number, rightfirstoffset, &lVis);
+
+ pfree(rightpage);
}
}
}
--
2.34.1
[application/octet-stream] v1-0001-Amcheck-optimize-speed-of-checking-unique-constra.patch (2.5K, ../../CALT9ZEFY-vsL4-cxta49wP=H7RjDaZk+Wg-ft4m-kVS_=WMsAQ@mail.gmail.com/6-v1-0001-Amcheck-optimize-speed-of-checking-unique-constra.patch)
download | inline diff:
From 2847e83ef91bfa3e3ebda6a8acd60bd835950716 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Thu, 25 Apr 2024 13:08:39 +0400
Subject: [PATCH v1 1/5] Amcheck: optimize speed of checking unique constraint
Check heap visibility only for non-equal non-deduplicated index
tuples. For deduplicated tuples we still need to check visibility
of all posting list entries because they represent equal
key values.
Reported-by: Noah Misch
---
contrib/amcheck/verify_nbtree.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 20da4a46ba..ae8012f15b 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -1428,6 +1428,7 @@ bt_target_page_check(BtreeCheckState *state)
BTScanInsert skey;
bool lowersizelimit;
ItemPointer scantid;
+ bool unique_checked = false;
CHECK_FOR_INTERRUPTS();
@@ -1774,10 +1775,14 @@ bt_target_page_check(BtreeCheckState *state)
* heap tuples visibility.
*/
if (state->checkunique && state->indexinfo->ii_Unique &&
- P_ISLEAF(topaque) && !skey->anynullkeys)
+ P_ISLEAF(topaque) && !skey->anynullkeys &&
+ (BTreeTupleIsPosting(itup) || ItemPointerIsValid(lVis_tid)))
+ {
bt_entry_unique_check(state, itup, state->targetblock, offset,
&lVis_i, &lVis_tid, &lVis_offset,
&lVis_block);
+ unique_checked = true;
+ }
if (state->checkunique && state->indexinfo->ii_Unique &&
P_ISLEAF(topaque) && OffsetNumberNext(offset) <= max)
@@ -1805,6 +1810,12 @@ bt_target_page_check(BtreeCheckState *state)
lVis_block = InvalidBlockNumber;
lVis_offset = InvalidOffsetNumber;
}
+ else if (!unique_checked)
+ {
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+ }
skey->scantid = scantid; /* Restore saved scan key state */
}
@@ -1889,6 +1900,11 @@ bt_target_page_check(BtreeCheckState *state)
/* The first key on the next page is the same */
if (_bt_compare(state->rel, rightkey, state->target, max) == 0 && !rightkey->anynullkeys)
{
+ if (!unique_checked)
+ bt_entry_unique_check(state, itup, state->targetblock, offset,
+ &lVis_i, &lVis_tid, &lVis_offset,
+ &lVis_block);
+
elog(DEBUG2, "cross page equal keys");
state->target = palloc_btree_page(state,
rightblock_number);
--
2.34.1
[application/octet-stream] v1-0005-Rename-checkunique-parameter-for-amcheck-and-pg_a.patch (20.8K, ../../CALT9ZEFY-vsL4-cxta49wP=H7RjDaZk+Wg-ft4m-kVS_=WMsAQ@mail.gmail.com/7-v1-0005-Rename-checkunique-parameter-for-amcheck-and-pg_a.patch)
download | inline diff:
From ff4d989173084431914220d1a66646a332fa9d71 Mon Sep 17 00:00:00 2001
From: Pavel Borisov <[email protected]>
Date: Thu, 25 Apr 2024 15:36:38 +0400
Subject: [PATCH v1 5/5] Rename checkunique parameter for amcheck and
pg_amcheck
Use more user friendly naming: --check-unique for pg_amcheck
command line, check_unique for amcheck sql functions
Reported-by: Peter Eisentraut
---
contrib/amcheck/amcheck--1.3--1.4.sql | 4 +--
contrib/amcheck/expected/check_btree.out | 12 +++----
contrib/amcheck/sql/check_btree.sql | 12 +++----
contrib/amcheck/verify_nbtree.c | 38 +++++++++++-----------
doc/src/sgml/amcheck.sgml | 8 ++---
doc/src/sgml/ref/pg_amcheck.sgml | 4 +--
src/bin/pg_amcheck/pg_amcheck.c | 20 ++++++------
src/bin/pg_amcheck/t/003_check.pl | 20 ++++++------
src/bin/pg_amcheck/t/005_opclass_damage.pl | 4 +--
9 files changed, 61 insertions(+), 61 deletions(-)
diff --git a/contrib/amcheck/amcheck--1.3--1.4.sql b/contrib/amcheck/amcheck--1.3--1.4.sql
index 75574eaa64..e0d4f92085 100644
--- a/contrib/amcheck/amcheck--1.3--1.4.sql
+++ b/contrib/amcheck/amcheck--1.3--1.4.sql
@@ -11,7 +11,7 @@
-- bt_index_parent_check()
--
CREATE FUNCTION bt_index_parent_check(index regclass,
- heapallindexed boolean, rootdescend boolean, checkunique boolean)
+ heapallindexed boolean, rootdescend boolean, check_unique boolean)
RETURNS VOID
AS 'MODULE_PATHNAME', 'bt_index_parent_check'
LANGUAGE C STRICT PARALLEL RESTRICTED;
@@ -19,7 +19,7 @@ LANGUAGE C STRICT PARALLEL RESTRICTED;
-- bt_index_check()
--
CREATE FUNCTION bt_index_check(index regclass,
- heapallindexed boolean, checkunique boolean)
+ heapallindexed boolean, check_unique boolean)
RETURNS VOID
AS 'MODULE_PATHNAME', 'bt_index_check'
LANGUAGE C STRICT PARALLEL RESTRICTED;
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.out
index e7fb5f5515..ebb91d5d78 100644
--- a/contrib/amcheck/expected/check_btree.out
+++ b/contrib/amcheck/expected/check_btree.out
@@ -200,25 +200,25 @@ SELECT bt_index_check('bttest_a_expr_idx', true);
(1 row)
-- UNIQUE constraint check
-SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, check_unique => true);
bt_index_check
----------------
(1 row)
-SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, check_unique => true);
bt_index_check
----------------
(1 row)
-SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, check_unique => true);
bt_index_parent_check
-----------------------
(1 row)
-SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, check_unique => true);
bt_index_parent_check
-----------------------
@@ -227,14 +227,14 @@ SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend
-- Check that null values in an unique index are not treated as equal
CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
-SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, check_unique => true);
bt_index_check
----------------
(1 row)
CREATE INDEX on bttest_unique_nulls (b,c);
-SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, check_unique => true);
bt_index_check
----------------
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sql
index 0793dbfeeb..f9ae33a6a4 100644
--- a/contrib/amcheck/sql/check_btree.sql
+++ b/contrib/amcheck/sql/check_btree.sql
@@ -136,17 +136,17 @@ CREATE INDEX bttest_a_expr_idx ON bttest_a ((ifun(id) + ifun(0)))
SELECT bt_index_check('bttest_a_expr_idx', true);
-- UNIQUE constraint check
-SELECT bt_index_check('bttest_a_idx', heapallindexed => true, checkunique => true);
-SELECT bt_index_check('bttest_b_idx', heapallindexed => false, checkunique => true);
-SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, checkunique => true);
-SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, checkunique => true);
+SELECT bt_index_check('bttest_a_idx', heapallindexed => true, check_unique => true);
+SELECT bt_index_check('bttest_b_idx', heapallindexed => false, check_unique => true);
+SELECT bt_index_parent_check('bttest_a_idx', heapallindexed => true, rootdescend => true, check_unique => true);
+SELECT bt_index_parent_check('bttest_b_idx', heapallindexed => true, rootdescend => false, check_unique => true);
-- Check that null values in an unique index are not treated as equal
CREATE TABLE bttest_unique_nulls (a serial, b int, c int UNIQUE);
INSERT INTO bttest_unique_nulls VALUES (generate_series(1, 10000), 2, default);
-SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_unique_nulls_c_key', heapallindexed => true, check_unique => true);
CREATE INDEX on bttest_unique_nulls (b,c);
-SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, checkunique => true);
+SELECT bt_index_check('bttest_unique_nulls_b_c_idx', heapallindexed => true, check_unique => true);
-- Check support of both 1B and 4B header sizes of short varlena datum
CREATE TABLE varlena_bug (v text);
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index dfc9ed769f..3b673bac95 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -83,7 +83,7 @@ typedef struct BtreeCheckState
/* Also making sure non-pivot tuples can be found by new search? */
bool rootdescend;
/* Also check uniqueness constraint if index is unique */
- bool checkunique;
+ bool check_unique;
/* Per-page context */
MemoryContext targetcontext;
/* Buffer access strategy */
@@ -159,12 +159,12 @@ PG_FUNCTION_INFO_V1(bt_index_parent_check);
static void bt_index_check_internal(Oid indrelid, bool parentcheck,
bool heapallindexed, bool rootdescend,
- bool checkunique);
+ bool check_unique);
static inline void btree_index_checkable(Relation rel);
static inline bool btree_index_mainfork_expected(Relation rel);
static void bt_check_every_level(Relation rel, Relation heaprel,
bool heapkeyspace, bool readonly, bool heapallindexed,
- bool rootdescend, bool checkunique);
+ bool rootdescend, bool check_unique);
static BtreeLevel bt_check_level_from_leftmost(BtreeCheckState *state,
BtreeLevel level);
static bool bt_leftmost_ignoring_half_dead(BtreeCheckState *state,
@@ -223,7 +223,7 @@ static inline ItemPointer BTreeTupleGetHeapTIDCareful(BtreeCheckState *state,
static inline ItemPointer BTreeTupleGetPointsToTID(IndexTuple itup);
/*
- * bt_index_check(index regclass, heapallindexed boolean, checkunique boolean)
+ * bt_index_check(index regclass, heapallindexed boolean, check_unique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -236,20 +236,20 @@ bt_index_check(PG_FUNCTION_ARGS)
{
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
- bool checkunique = false;
+ bool check_unique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
if (PG_NARGS() == 3)
- checkunique = PG_GETARG_BOOL(2);
+ check_unique = PG_GETARG_BOOL(2);
- bt_index_check_internal(indrelid, false, heapallindexed, false, checkunique);
+ bt_index_check_internal(indrelid, false, heapallindexed, false, check_unique);
PG_RETURN_VOID();
}
/*
- * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean)
+ * bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, check_unique boolean)
*
* Verify integrity of B-Tree index.
*
@@ -263,16 +263,16 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
Oid indrelid = PG_GETARG_OID(0);
bool heapallindexed = false;
bool rootdescend = false;
- bool checkunique = false;
+ bool check_unique = false;
if (PG_NARGS() >= 2)
heapallindexed = PG_GETARG_BOOL(1);
if (PG_NARGS() >= 3)
rootdescend = PG_GETARG_BOOL(2);
if (PG_NARGS() == 4)
- checkunique = PG_GETARG_BOOL(3);
+ check_unique = PG_GETARG_BOOL(3);
- bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, checkunique);
+ bt_index_check_internal(indrelid, true, heapallindexed, rootdescend, check_unique);
PG_RETURN_VOID();
}
@@ -282,7 +282,7 @@ bt_index_parent_check(PG_FUNCTION_ARGS)
*/
static void
bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
- bool rootdescend, bool checkunique)
+ bool rootdescend, bool check_unique)
{
Oid heapid;
Relation indrel;
@@ -394,7 +394,7 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
/* Check index, possibly against table it is an index on */
bt_check_every_level(indrel, heaprel, heapkeyspace, parentcheck,
- heapallindexed, rootdescend, checkunique);
+ heapallindexed, rootdescend, check_unique);
}
/* Roll back any GUC changes executed by index functions */
@@ -496,7 +496,7 @@ btree_index_mainfork_expected(Relation rel)
static void
bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
bool readonly, bool heapallindexed, bool rootdescend,
- bool checkunique)
+ bool check_unique)
{
BtreeCheckState *state;
Page metapage;
@@ -528,7 +528,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
state->readonly = readonly;
state->heapallindexed = heapallindexed;
state->rootdescend = rootdescend;
- state->checkunique = checkunique;
+ state->check_unique = check_unique;
state->snapshot = InvalidSnapshot;
if (state->heapallindexed)
@@ -592,7 +592,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
* performance take it once per index check. If snapshot already taken
* reuse it.
*/
- if (state->checkunique)
+ if (state->check_unique)
{
state->indexinfo = BuildIndexInfo(state->rel);
if (state->indexinfo->ii_Unique)
@@ -1770,7 +1770,7 @@ bt_target_page_check(BtreeCheckState *state)
* If the index is unique verify entries uniqueness by checking the
* heap tuples visibility.
*/
- if (state->checkunique && state->indexinfo->ii_Unique &&
+ if (state->check_unique && state->indexinfo->ii_Unique &&
P_ISLEAF(topaque) && !skey->anynullkeys &&
(BTreeTupleIsPosting(itup) || ItemPointerIsValid(lVis.tid)))
{
@@ -1778,7 +1778,7 @@ bt_target_page_check(BtreeCheckState *state)
unique_checked = true;
}
- if (state->checkunique && state->indexinfo->ii_Unique &&
+ if (state->check_unique && state->indexinfo->ii_Unique &&
P_ISLEAF(topaque) && OffsetNumberNext(offset) <= max)
{
/* Save current scankey tid */
@@ -1873,7 +1873,7 @@ bt_target_page_check(BtreeCheckState *state)
* If index has unique constraint make sure that no more than one
* found equal items is visible.
*/
- if (state->checkunique && state->indexinfo->ii_Unique &&
+ if (state->check_unique && state->indexinfo->ii_Unique &&
rightkey && P_ISLEAF(topaque) && !P_RIGHTMOST(topaque))
{
BlockNumber rightblock_number = topaque->btpo_next;
diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml
index 3af065615b..3464ab5e76 100644
--- a/doc/src/sgml/amcheck.sgml
+++ b/doc/src/sgml/amcheck.sgml
@@ -61,7 +61,7 @@
<variablelist>
<varlistentry>
<term>
- <function>bt_index_check(index regclass, heapallindexed boolean, checkunique boolean) returns void</function>
+ <function>bt_index_check(index regclass, heapallindexed boolean, check_unique boolean) returns void</function>
<indexterm>
<primary>bt_index_check</primary>
</indexterm>
@@ -118,7 +118,7 @@ ORDER BY c.relpages DESC LIMIT 10;
that span child/parent relationships, but will verify the
presence of all heap tuples as index tuples within the index
when <parameter>heapallindexed</parameter> is
- <literal>true</literal>. When <parameter>checkunique</parameter>
+ <literal>true</literal>. When <parameter>check_unique</parameter>
is <literal>true</literal> <function>bt_index_check</function> will
check that no more than one among duplicate entries in unique
index is visible. When a routine, lightweight test for
@@ -132,7 +132,7 @@ ORDER BY c.relpages DESC LIMIT 10;
<varlistentry>
<term>
- <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, checkunique boolean) returns void</function>
+ <function>bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean, check_unique boolean) returns void</function>
<indexterm>
<primary>bt_index_parent_check</primary>
</indexterm>
@@ -145,7 +145,7 @@ ORDER BY c.relpages DESC LIMIT 10;
Optionally, when the <parameter>heapallindexed</parameter>
argument is <literal>true</literal>, the function verifies the
presence of all heap tuples that should be found within the
- index. When <parameter>checkunique</parameter>
+ index. When <parameter>check_unique</parameter>
is <literal>true</literal> <function>bt_index_parent_check</function> will
check that no more than one among duplicate entries in unique
index is visible. When the optional <parameter>rootdescend</parameter>
diff --git a/doc/src/sgml/ref/pg_amcheck.sgml b/doc/src/sgml/ref/pg_amcheck.sgml
index 067c806b46..0837045632 100644
--- a/doc/src/sgml/ref/pg_amcheck.sgml
+++ b/doc/src/sgml/ref/pg_amcheck.sgml
@@ -434,12 +434,12 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>--checkunique</option></term>
+ <term><option>--check-unique</option></term>
<listitem>
<para>
For each index with unique constraint checked, verify that no more than
one among duplicate entries is visible in the index using <xref linkend="amcheck"/>'s
- <option>checkunique</option> option.
+ <option>check-unique</option> option.
</para>
</listitem>
</varlistentry>
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 7e3101704d..3aaf69a3b4 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -102,7 +102,7 @@ typedef struct AmcheckOptions
bool parent_check;
bool rootdescend;
bool heapallindexed;
- bool checkunique;
+ bool check_unique;
/* heap and btree hybrid option */
bool no_btree_expansion;
@@ -133,7 +133,7 @@ static AmcheckOptions opts = {
.parent_check = false,
.rootdescend = false,
.heapallindexed = false,
- .checkunique = false,
+ .check_unique = false,
.no_btree_expansion = false
};
@@ -270,7 +270,7 @@ main(int argc, char *argv[])
{"heapallindexed", no_argument, NULL, 11},
{"parent-check", no_argument, NULL, 12},
{"install-missing", optional_argument, NULL, 13},
- {"checkunique", no_argument, NULL, 14},
+ {"check-unique", no_argument, NULL, 14},
{NULL, 0, NULL, 0}
};
@@ -439,7 +439,7 @@ main(int argc, char *argv[])
opts.install_schema = pg_strdup(optarg);
break;
case 14:
- opts.checkunique = true;
+ opts.check_unique = true;
break;
default:
/* getopt_long already emitted a complaint */
@@ -602,7 +602,7 @@ main(int argc, char *argv[])
* constraint check with warning if it is not yet supported by
* amcheck.
*/
- if (opts.checkunique == true)
+ if (opts.check_unique == true)
{
/*
* Now amcheck has only major and minor versions in the string but
@@ -617,11 +617,11 @@ main(int argc, char *argv[])
sscanf(amcheck_version, "%d.%d.%d", &vmaj, &vmin, &vrev);
/*
- * checkunique option is supported in amcheck since version 1.4
+ * check_unique option is supported in amcheck since version 1.4
*/
if ((vmaj == 1 && vmin < 4) || vmaj == 0)
{
- pg_log_warning("--checkunique option is not supported by amcheck "
+ pg_log_warning("--check-unique option is not supported by amcheck "
"version \"%s\"", amcheck_version);
dat->is_checkunique = false;
}
@@ -895,7 +895,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
(opts.rootdescend ? "true" : "false"),
- (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
+ (rel->datinfo->is_checkunique ? ", check_unique := true" : ""),
rel->reloid);
else
appendPQExpBuffer(sql,
@@ -909,7 +909,7 @@ prepare_btree_command(PQExpBuffer sql, RelationInfo *rel, PGconn *conn)
"AND i.indisready AND i.indisvalid AND i.indislive",
rel->datinfo->amcheck_schema,
(opts.heapallindexed ? "true" : "false"),
- (rel->datinfo->is_checkunique ? ", checkunique := true" : ""),
+ (rel->datinfo->is_checkunique ? ", check_unique := true" : ""),
rel->reloid);
}
@@ -1208,7 +1208,7 @@ help(const char *progname)
printf(_(" --heapallindexed check that all heap tuples are found within indexes\n"));
printf(_(" --parent-check check index parent/child relationships\n"));
printf(_(" --rootdescend search from root page to refind tuples\n"));
- printf(_(" --checkunique check unique constraint if index is unique\n"));
+ printf(_(" --check-unique check unique constraint if index is unique\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port\n"));
diff --git a/src/bin/pg_amcheck/t/003_check.pl b/src/bin/pg_amcheck/t/003_check.pl
index 4b16bda6a4..0a8cf8bced 100644
--- a/src/bin/pg_amcheck/t/003_check.pl
+++ b/src/bin/pg_amcheck/t/003_check.pl
@@ -523,35 +523,35 @@ $node->command_checks_all(
$node->command_checks_all(
[
@cmd, '-s', 's1', '-i', 't1_btree', '--parent-check',
- '--checkunique', 'db1'
+ '--check-unique', 'db1'
],
2,
[$index_missing_relation_fork_re],
[$no_output_re],
- 'pg_amcheck smoke test --parent-check --checkunique');
+ 'pg_amcheck smoke test --parent-check --check-unique');
$node->command_checks_all(
[
@cmd, '-s', 's1', '-i', 't1_btree', '--heapallindexed',
- '--rootdescend', '--checkunique', 'db1'
+ '--rootdescend', '--check-unique', 'db1'
],
2,
[$index_missing_relation_fork_re],
[$no_output_re],
- 'pg_amcheck smoke test --heapallindexed --rootdescend --checkunique');
+ 'pg_amcheck smoke test --heapallindexed --rootdescend --check-unique');
$node->command_checks_all(
[
- @cmd, '--checkunique', '-d', 'db1', '-d', 'db2',
+ @cmd, '--check-unique', '-d', 'db1', '-d', 'db2',
'-d', 'db3', '-S', 's*'
],
0,
[$no_output_re],
[$no_output_re],
- 'pg_amcheck excluding all corrupt schemas with --checkunique option');
+ 'pg_amcheck excluding all corrupt schemas with --check-unique option');
#
-# Smoke test for checkunique option for not supported versions.
+# Smoke test for check-unique option for not supported versions.
#
$node->safe_psql(
'db3', q(
@@ -560,11 +560,11 @@ $node->safe_psql(
));
$node->command_checks_all(
- [ @cmd, '--checkunique', 'db3' ],
+ [ @cmd, '--check-unique', 'db3' ],
0,
[$no_output_re],
[
- qr/pg_amcheck: warning: --checkunique option is not supported by amcheck version "1.3"/
+ qr/pg_amcheck: warning: --check-unique option is not supported by amcheck version "1.3"/
],
- 'pg_amcheck smoke test --checkunique');
+ 'pg_amcheck smoke test --check-unique');
done_testing();
diff --git a/src/bin/pg_amcheck/t/005_opclass_damage.pl b/src/bin/pg_amcheck/t/005_opclass_damage.pl
index 1eea215227..0db58dee5b 100644
--- a/src/bin/pg_amcheck/t/005_opclass_damage.pl
+++ b/src/bin/pg_amcheck/t/005_opclass_damage.pl
@@ -90,7 +90,7 @@ $node->safe_psql(
# We should get no corruptions
$node->command_like(
- [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ [ 'pg_amcheck', '--check-unique', '-p', $node->port, 'postgres' ],
qr/^$/,
'pg_amcheck all schemas, tables and indexes reports no corruption');
@@ -116,7 +116,7 @@ $node->safe_psql(
# Unique index corruption should now be reported
$node->command_checks_all(
- [ 'pg_amcheck', '--checkunique', '-p', $node->port, 'postgres' ],
+ [ 'pg_amcheck', '--check-unique', '-p', $node->port, 'postgres' ],
2,
[qr/index uniqueness is violated for index "bttest_unique_idx"/],
[],
--
2.34.1
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-25 12:59 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2024-04-25 13:44 ` Karina Litskevich <[email protected]>
2024-04-25 13:54 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Karina Litskevich @ 2024-04-25 13:44 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Peter Eisentraut <[email protected]>; Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
Hi, hackers!
On Thu, Apr 25, 2024 at 4:00 PM Pavel Borisov <[email protected]>
wrote:
> 0005: Rename checkunique parameter to more user friendly as proposed by
> Peter Eisentraut and Alexander Korotkov
>
I'm not sure renaming checkunique is a good idea. Other arguments of
bt_index_check and bt_index_parent_check functions (heapallindexed and
rootdescend) don't have underscore character in them. Corresponding
pg_amcheck options (--heapallindexed and --rootdescend) are also written
in one piece. check_unique and --check-unique stand out. Making arguments
and options in different styles doesn't seem user friendly to me.
Best regards,
Karina Litskevich
Postgres Professional: http://postgrespro.com/
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-25 12:59 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2024-04-25 13:44 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Karina Litskevich <[email protected]>
@ 2024-04-25 13:54 ` Pavel Borisov <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Pavel Borisov @ 2024-04-25 13:54 UTC (permalink / raw)
To: Karina Litskevich <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Peter Eisentraut <[email protected]>; Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
Hi, Karina!
On Thu, 25 Apr 2024 at 17:44, Karina Litskevich <[email protected]>
wrote:
> Hi, hackers!
>
> On Thu, Apr 25, 2024 at 4:00 PM Pavel Borisov <[email protected]>
> wrote:
>
>> 0005: Rename checkunique parameter to more user friendly as proposed by
>> Peter Eisentraut and Alexander Korotkov
>>
>
> I'm not sure renaming checkunique is a good idea. Other arguments of
> bt_index_check and bt_index_parent_check functions (heapallindexed and
> rootdescend) don't have underscore character in them. Corresponding
> pg_amcheck options (--heapallindexed and --rootdescend) are also written
> in one piece. check_unique and --check-unique stand out. Making arguments
> and options in different styles doesn't seem user friendly to me.
>
I did it under the consensus of Peter Eisentraut and Alexander Korotkov.
The pro for renaming is more user-friendly naming, I also agree.
The cons is that we already have both styles: "non-user friendly"
heapallindexed and rootdescend and "user-friendly" parent-check.
I'm ready to go with consensus in this matter. It's also not yet too late
to make it unique-check (instead of check-unique) to be better in style
with parent-check.
Kind regards,
Pavel
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-25 12:59 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2024-05-01 02:24 ` Noah Misch <[email protected]>
2024-05-01 02:26 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Noah Misch @ 2024-05-01 02:24 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Peter Eisentraut <[email protected]>; Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
On Thu, Apr 25, 2024 at 04:59:54PM +0400, Pavel Borisov wrote:
> 0001: Optimize speed by avoiding heap visibility checking for different
> non-deduplicated index tuples as proposed by Noah Misch
>
> Speed measurements on my laptop using the exact method recommended by Noah
> upthread:
> Current master branch: checkunique off: 144s, checkunique on: 419s
> With patch 0001: checkunique off: 141s, checkunique on: 171s
Where is the CPU time going to make it still be 21% slower w/ checkunique on?
It's a great improvement vs. current master, but I don't have an obvious
explanation for the remaining +21%.
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-01 20:23 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-03-02 14:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-04-08 14:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. David Steele <[email protected]>
2021-12-20 15:37 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-22 08:01 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-12-23 18:05 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Максим Орлов <[email protected]>
2022-02-21 14:14 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Greg Stark <[email protected]>
2022-04-04 09:18 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2022-09-22 15:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Andres Freund <[email protected]>
2022-09-27 08:04 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-17 06:38 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Alexander Korotkov <[email protected]>
2024-04-25 12:59 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2024-05-01 02:24 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Noah Misch <[email protected]>
@ 2024-05-01 02:26 ` Alexander Korotkov <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Alexander Korotkov @ 2024-05-01 02:26 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Peter Eisentraut <[email protected]>; Aleksander Alekseev <[email protected]>; Postgres hackers <[email protected]>; Maxim Orlov <[email protected]>; Andres Freund <[email protected]>; Greg Stark <[email protected]>; Julien Rouhaud <[email protected]>; Mark Dilger <[email protected]>; David Steele <[email protected]>; Peter Geoghegan <[email protected]>; Maxim Orlov <[email protected]>; [email protected]
Hi Noah,
On Wed, May 1, 2024 at 5:24 AM Noah Misch <[email protected]> wrote:
> On Thu, Apr 25, 2024 at 04:59:54PM +0400, Pavel Borisov wrote:
> > 0001: Optimize speed by avoiding heap visibility checking for different
> > non-deduplicated index tuples as proposed by Noah Misch
> >
> > Speed measurements on my laptop using the exact method recommended by Noah
> > upthread:
> > Current master branch: checkunique off: 144s, checkunique on: 419s
> > With patch 0001: checkunique off: 141s, checkunique on: 171s
>
> Where is the CPU time going to make it still be 21% slower w/ checkunique on?
> It's a great improvement vs. current master, but I don't have an obvious
> explanation for the remaining +21%.
I think there is at least extra index tuples comparison.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
2021-02-09 18:43 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-01 19:21 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Mark Dilger <[email protected]>
@ 2021-03-03 02:54 ` Peter Geoghegan <[email protected]>
2 siblings, 0 replies; 45+ messages in thread
From: Peter Geoghegan @ 2021-03-03 02:54 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Anastasia Lubennikova <[email protected]>; Postgres hackers <[email protected]>
On Mon, Mar 1, 2021 at 11:22 AM Mark Dilger
<[email protected]> wrote:
> If bt_index_check() and bt_index_parent_check() are to have this functionality, shouldn't there be an option controlling it much as the option (heapallindexed boolean) controls checking whether all entries in the heap are indexed in the btree? It seems inconsistent to have an option to avoid checking the heap for that, but not for this.
I agree. Actually, it should probably use the same snapshot as the
heapallindexed=true case. So either only perform unique constraint
verification when that option is used, or invent a new option that
will still share the snapshot used by heapallindexed=true (when the
options are combined).
> The regression test you provided is not portable. I am getting lots of errors due to differing output of the form "page lsn=0/4DAD7E0". You might turn this into a TAP test and use a regular expression to check the output.
I would test this using a custom opclass that does simple fault
injection. For example, an opclass that indexes integers, but can be
configured to dynamically make 0 values equal or unequal to each
other. That's more representative of real-world problems.
You "break the warranty" by updating pg_index, even compared to
updating other system catalogs. In particular, you break the
"indcheckxmin wait -- wait for xmin to be old before using index"
stuff in get_relation_info(). So it seems worse than updating
pg_attribute, for example (which is something that the tests do
already).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
@ 2021-03-03 03:10 ` Peter Geoghegan <[email protected]>
2021-03-03 08:08 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2 siblings, 1 reply; 45+ messages in thread
From: Peter Geoghegan @ 2021-03-03 03:10 UTC (permalink / raw)
To: Pavel Borisov <[email protected]>; +Cc: Postgres hackers <[email protected]>
On Mon, Feb 8, 2021 at 2:46 AM Pavel Borisov <[email protected]> wrote:
> Caveat: if the first entry on the next index page has a key equal to the key on a previous page AND all heap tid's corresponding to this entry are invisible, currently cross-page check can not detect unique constraint violation between previous index page entry and 2nd, 3d and next current index page entries. In this case, there would be a message that recommends doing VACUUM to remove the invisible entries from the index and repeat the check. (Generally, it is recommended to do vacuum before the check, but for the testing purpose I'd recommend turning it off to check the detection of visible-invisible-visible duplicates scenarios)
It's rather unlikely that equal values in a unique index will end up
on different leaf pages. It's really rare, in fact. This following
comment block from nbtinsert.c (which appears right before we call
_bt_check_unique()) explains why this is:
* It might be necessary to check a page to the right in _bt_check_unique,
* though that should be very rare. In practice the first page the value ...
You're going to have to "couple" buffer locks in the style of
_bt_check_unique() (as well as keeping a buffer lock on "the first
leaf page a duplicate might be on" throughout) if you need the test to
work reliably. But why bother with that? The tool doesn't have to be
100% perfect at detecting corruption (nothing can be), and it's rather
unlikely that it will matter for this test. A simple test that doesn't
handle cross-page duplicates is still going to be very effective.
I don't think that it's acceptable for your new check to raise a
WARNING instead of an ERROR. I especially don't like that the new
unique checking functionality merely warns that the index *might* be
corrupt. False positives are always unacceptable within amcheck, and I
think that this is a false positive.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-03-03 03:10 ` Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Peter Geoghegan <[email protected]>
@ 2021-03-03 08:08 ` Pavel Borisov <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Pavel Borisov @ 2021-03-03 08:08 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: Postgres hackers <[email protected]>
>
> You're going to have to "couple" buffer locks in the style of
> _bt_check_unique() (as well as keeping a buffer lock on "the first
> leaf page a duplicate might be on" throughout) if you need the test to
> work reliably. But why bother with that? The tool doesn't have to be
> 100% perfect at detecting corruption (nothing can be), and it's rather
> unlikely that it will matter for this test. A simple test that doesn't
> handle cross-page duplicates is still going to be very effective.
>
Indeed at first, I did the test which doesn't bother checking duplicates
cross-page which I considered very rare, but then a customer sent me his
corrupted index where I found this rare thing which was not detectable by
amcheck and he was puzzled with the issue. Even rare inconsistencies can
appear when people handle huge amounts of data. So I did an update that
handles a wider class of errors. I don't suppose that cross page unique
check is expensive as it uses same things that are already used in amcheck
for cross-page checks.
Is it suitable if I omit suspected duplicates message in the very-very rare
case amcheck can not detect but leave cross-page checks?
>>I don't think that it's acceptable for your new check to raise a
>>WARNING instead of an ERROR.
It is not instead of an ERROR. If at least one violation is detected,
amcheck will output the final ERROR message. The purpose is not to stop
checking at the first violation. But I can make them reported in a current
amcheck style if it is necessary.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH] xlog_block_info: show compression method
@ 2022-02-19 17:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Justin Pryzby @ 2022-02-19 17:35 UTC (permalink / raw)
---
src/backend/access/transam/xlogrecovery.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 4ad145dd167..75a72b47f78 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -2238,7 +2238,20 @@ xlog_block_info(StringInfo buf, XLogReaderState *record)
rlocator.relNumber,
blk);
if (XLogRecHasBlockImage(record, block_id))
- appendStringInfoString(buf, " FPW");
+ {
+ if (!BKPIMAGE_COMPRESSED(record->blocks[block_id].bimg_info))
+ appendStringInfoString(buf, " FPW");
+ else
+ {
+ uint8 info = record->blocks[block_id].bimg_info;
+ char *compression =
+ (BKPIMAGE_COMPRESS_PGLZ & info) ? "pglz" :
+ (BKPIMAGE_COMPRESS_LZ4 & info) ? "lz4" :
+ (BKPIMAGE_COMPRESS_ZSTD & info) ? "zstd" : "unknown";
+
+ appendStringInfo(buf, " FPW, compression method: %s", compression);
+ }
+ }
}
}
--
2.17.1
--um2V5WpqCyd73IVb--
^ permalink raw reply [nested|flat] 45+ messages in thread
end of thread, other threads:[~2024-05-01 02:26 UTC | newest]
Thread overview: 45+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-08 10:46 [PATCH] Improve amcheck to also check UNIQUE constraint in btree index. Pavel Borisov <[email protected]>
2021-02-08 19:31 ` Pavel Borisov <[email protected]>
2021-02-08 21:46 ` Mark Dilger <[email protected]>
2021-02-09 18:43 ` Pavel Borisov <[email protected]>
2021-02-09 18:57 ` Pavel Borisov <[email protected]>
2021-02-09 19:41 ` Zhihong Yu <[email protected]>
2021-03-01 19:21 ` Mark Dilger <[email protected]>
2021-03-01 20:05 ` Pavel Borisov <[email protected]>
2021-03-01 20:20 ` Mark Dilger <[email protected]>
2021-03-01 20:23 ` Tom Lane <[email protected]>
2021-03-01 20:23 ` Pavel Borisov <[email protected]>
2021-03-01 21:05 ` Mark Dilger <[email protected]>
2021-03-02 14:08 ` Pavel Borisov <[email protected]>
2021-03-15 15:11 ` Mark Dilger <[email protected]>
2021-04-08 14:36 ` David Steele <[email protected]>
2021-12-20 15:37 ` Pavel Borisov <[email protected]>
2021-12-20 18:02 ` Mark Dilger <[email protected]>
2021-12-22 08:01 ` Pavel Borisov <[email protected]>
2021-12-23 16:31 ` Mark Dilger <[email protected]>
2021-12-23 18:05 ` Максим Орлов <[email protected]>
2022-02-21 14:14 ` Maxim Orlov <[email protected]>
2022-04-03 00:02 ` Greg Stark <[email protected]>
2022-04-04 09:18 ` Pavel Borisov <[email protected]>
2022-05-11 13:04 ` Pavel Borisov <[email protected]>
2022-05-20 13:46 ` Pavel Borisov <[email protected]>
2022-07-20 14:15 ` Aleksander Alekseev <[email protected]>
2022-09-07 13:44 ` Dmitry Koval <[email protected]>
2022-09-08 13:29 ` Karina Litskevich <[email protected]>
2022-09-22 15:13 ` Andres Freund <[email protected]>
2022-09-27 08:04 ` Maxim Orlov <[email protected]>
2022-09-28 08:36 ` Maxim Orlov <[email protected]>
2022-09-28 08:43 ` Aleksander Alekseev <[email protected]>
2023-10-24 20:13 ` Alexander Korotkov <[email protected]>
2023-10-30 07:29 ` Pavel Borisov <[email protected]>
2024-04-17 06:38 ` Peter Eisentraut <[email protected]>
2024-04-24 09:58 ` Alexander Korotkov <[email protected]>
2024-04-25 12:59 ` Pavel Borisov <[email protected]>
2024-04-25 13:44 ` Karina Litskevich <[email protected]>
2024-04-25 13:54 ` Pavel Borisov <[email protected]>
2024-05-01 02:24 ` Noah Misch <[email protected]>
2024-05-01 02:26 ` Alexander Korotkov <[email protected]>
2021-03-03 02:54 ` Peter Geoghegan <[email protected]>
2021-03-03 03:10 ` Peter Geoghegan <[email protected]>
2021-03-03 08:08 ` Pavel Borisov <[email protected]>
2022-02-19 17:35 [PATCH] xlog_block_info: show compression method Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox