agora inbox for [email protected]help / color / mirror / Atom feed
Thinko/typo in ExecSimpleRelationInsert 24+ messages / 5 participants [nested] [flat]
* Thinko/typo in ExecSimpleRelationInsert @ 2018-06-26 09:42 Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Ashutosh Bapat @ 2018-06-26 09:42 UTC (permalink / raw) To: pgsql-hackers Hi, There seems to be a thinko/typo in ExecSimpleRelationInsert(). A tuple can never store a slot, but a comment in that function says so. Tried to fix it in the patch attached. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company Attachments: [text/x-patch] 0002-Fix-a-thinko-typo-in-ExecSimpleRelationInsert.patch (1000B, ../../CAFjFpRcHhNhXdegyJv3KKDWrwO1_NB_KYZM_ZSDeMOZaL1A5jQ@mail.gmail.com/2-0002-Fix-a-thinko-typo-in-ExecSimpleRelationInsert.patch) download | inline diff: From f3d005064d8d8cf366b021721c7a5b3d58554576 Mon Sep 17 00:00:00 2001 From: Ashutosh Bapat <[email protected]> Date: Tue, 26 Jun 2018 15:02:07 +0530 Subject: [PATCH 2/2] Fix a thinko/typo in ExecSimpleRelationInsert() A slot can not be stored in a tuple but it's vice versa. --- src/backend/executor/execReplication.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 4fbdfc0..2109e45 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -415,7 +415,7 @@ ExecSimpleRelationInsert(EState *estate, TupleTableSlot *slot) if (rel->rd_att->constr) ExecConstraints(resultRelInfo, slot, estate, true); - /* Store the slot into tuple that we can inspect. */ + /* Materialize slot into a tuple that we can inspect. */ tuple = ExecMaterializeSlot(slot); /* OK, store the tuple and create index entries for it */ -- 2.7.4 ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> @ 2018-06-26 11:03 ` Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Ashutosh Bapat @ 2018-06-26 11:03 UTC (permalink / raw) To: pgsql-hackers Looks like we need similar adjustment in ExecSimpleRelationUpdate() as well. Updated the patch. On Tue, Jun 26, 2018 at 3:12 PM, Ashutosh Bapat <[email protected]> wrote: > Hi, > There seems to be a thinko/typo in ExecSimpleRelationInsert(). A tuple > can never store a slot, but a comment in that function says so. Tried > to fix it in the patch attached. > > -- > Best Wishes, > Ashutosh Bapat > EnterpriseDB Corporation > The Postgres Database Company -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company Attachments: [text/x-patch] 0002-Fix-a-thinko-typo-in-ExecSimpleRelationInsert-Update.patch (1.4K, ../../CAFjFpRdA6j6vfBNxrDPtHotyo2+zY=odRZWENoH_7Zs6h2pJWQ@mail.gmail.com/2-0002-Fix-a-thinko-typo-in-ExecSimpleRelationInsert-Update.patch) download | inline diff: From 387148433d33eb7a9c2960e9ba554b1e28d04ef1 Mon Sep 17 00:00:00 2001 From: Ashutosh Bapat <[email protected]> Date: Tue, 26 Jun 2018 15:02:07 +0530 Subject: [PATCH 2/2] Fix a thinko/typo in ExecSimpleRelationInsert/Update A slot can not be stored in a tuple but it's vice versa. --- src/backend/executor/execReplication.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 4fbdfc0..ee6ad37 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -415,7 +415,7 @@ ExecSimpleRelationInsert(EState *estate, TupleTableSlot *slot) if (rel->rd_att->constr) ExecConstraints(resultRelInfo, slot, estate, true); - /* Store the slot into tuple that we can inspect. */ + /* Materialize slot into a tuple that we can inspect. */ tuple = ExecMaterializeSlot(slot); /* OK, store the tuple and create index entries for it */ @@ -480,7 +480,7 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate, if (rel->rd_att->constr) ExecConstraints(resultRelInfo, slot, estate, true); - /* Store the slot into tuple that we can write. */ + /* Materialize slot into a tuple that we can inspect. */ tuple = ExecMaterializeSlot(slot); /* OK, update the tuple and index entries for it */ -- 2.7.4 ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> @ 2018-06-26 12:48 ` Amit Kapila <[email protected]> 2018-06-26 13:32 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Amit Kapila @ 2018-06-26 12:48 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers On Tue, Jun 26, 2018 at 4:33 PM, Ashutosh Bapat <[email protected]> wrote: > Looks like we need similar adjustment in ExecSimpleRelationUpdate() as > well. Updated the patch. > - /* Store the slot into tuple that we can write. */ + /* Materialize slot into a tuple that we can inspect. */ tuple = ExecMaterializeSlot(slot); I think it is better to keep the last word of the sentence as "write" instead of "inspect" as was in the original sentence. It makes more sense as we are materializing the tuple to write it. Similarly, in the other change in the patch can use "write". -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> @ 2018-06-26 13:32 ` Ashutosh Bapat <[email protected]> 2018-06-26 15:13 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Ashutosh Bapat @ 2018-06-26 13:32 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: pgsql-hackers On Tue, Jun 26, 2018 at 6:18 PM, Amit Kapila <[email protected]> wrote: > On Tue, Jun 26, 2018 at 4:33 PM, Ashutosh Bapat > <[email protected]> wrote: >> Looks like we need similar adjustment in ExecSimpleRelationUpdate() as >> well. Updated the patch. >> > > - /* Store the slot into tuple that we can write. */ > + /* Materialize slot into a tuple that we can inspect. */ > tuple = ExecMaterializeSlot(slot); > > I think it is better to keep the last word of the sentence as "write" > instead of "inspect" as was in the original sentence. A copy-pasto while correcting a typo :) > It makes more > sense as we are materializing the tuple to write it. Similarly, in > the other change in the patch can use "write". Are you suggesting that we should use "write" in the modified comment instead of "inspect" in original comment. Ok, I have now corrected grammar as well. Here's updated patch. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company Attachments: [text/x-patch] 0002-Fix-a-thinko-typo-in-ExecSimpleRelationInsert-Update.patch (1.4K, ../../CAFjFpRewd8RN7OxZFn5C1v_zLNJEYjeQY+6OfX2g4F+vPuXJug@mail.gmail.com/2-0002-Fix-a-thinko-typo-in-ExecSimpleRelationInsert-Update.patch) download | inline diff: From fd22bc6d7d786680eb490a99da136a5dc8ba2c1b Mon Sep 17 00:00:00 2001 From: Ashutosh Bapat <[email protected]> Date: Tue, 26 Jun 2018 15:02:07 +0530 Subject: [PATCH 2/2] Fix a thinko/typo in ExecSimpleRelationInsert/Update A slot can not be stored in a tuple but it's vice versa. --- src/backend/executor/execReplication.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 4fbdfc0..74b3199 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -415,7 +415,7 @@ ExecSimpleRelationInsert(EState *estate, TupleTableSlot *slot) if (rel->rd_att->constr) ExecConstraints(resultRelInfo, slot, estate, true); - /* Store the slot into tuple that we can inspect. */ + /* Materialize slot into a tuple that we can scribble upon. */ tuple = ExecMaterializeSlot(slot); /* OK, store the tuple and create index entries for it */ @@ -480,7 +480,7 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate, if (rel->rd_att->constr) ExecConstraints(resultRelInfo, slot, estate, true); - /* Store the slot into tuple that we can write. */ + /* Materialize slot into a tuple that we can scribble upon. */ tuple = ExecMaterializeSlot(slot); /* OK, update the tuple and index entries for it */ -- 2.7.4 ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-26 13:32 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> @ 2018-06-26 15:13 ` Amit Kapila <[email protected]> 2018-06-27 04:39 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Amit Kapila @ 2018-06-26 15:13 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers On Tue, Jun 26, 2018 at 7:02 PM, Ashutosh Bapat <[email protected]> wrote: > On Tue, Jun 26, 2018 at 6:18 PM, Amit Kapila <[email protected]> wrote: >> On Tue, Jun 26, 2018 at 4:33 PM, Ashutosh Bapat >> <[email protected]> wrote: >>> Looks like we need similar adjustment in ExecSimpleRelationUpdate() as >>> well. Updated the patch. >>> >> >> - /* Store the slot into tuple that we can write. */ >> + /* Materialize slot into a tuple that we can inspect. */ >> tuple = ExecMaterializeSlot(slot); >> >> I think it is better to keep the last word of the sentence as "write" >> instead of "inspect" as was in the original sentence. > > A copy-pasto while correcting a typo :) > >> It makes more >> sense as we are materializing the tuple to write it. Similarly, in >> the other change in the patch can use "write". > > Are you suggesting that we should use "write" in the modified comment > instead of "inspect" in original comment. > Yes. Another variant used in few other similar places is: /* * get the heap tuple out of the tuple table slot, making sure we have a * writable copy */ > Ok, I have now corrected grammar as well. Here's updated patch. > Your new comment is also correct. I like the comment already used in code as that makes the code consistent, what do you think? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-26 13:32 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 15:13 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> @ 2018-06-27 04:39 ` Ashutosh Bapat <[email protected]> 2018-06-27 05:54 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Ashutosh Bapat @ 2018-06-27 04:39 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: pgsql-hackers On Tue, Jun 26, 2018 at 8:43 PM, Amit Kapila <[email protected]> wrote: > On Tue, Jun 26, 2018 at 7:02 PM, Ashutosh Bapat > <[email protected]> wrote: >> On Tue, Jun 26, 2018 at 6:18 PM, Amit Kapila <[email protected]> wrote: >>> On Tue, Jun 26, 2018 at 4:33 PM, Ashutosh Bapat >>> <[email protected]> wrote: >>>> Looks like we need similar adjustment in ExecSimpleRelationUpdate() as >>>> well. Updated the patch. >>>> >>> >>> - /* Store the slot into tuple that we can write. */ >>> + /* Materialize slot into a tuple that we can inspect. */ >>> tuple = ExecMaterializeSlot(slot); >>> >>> I think it is better to keep the last word of the sentence as "write" >>> instead of "inspect" as was in the original sentence. >> >> A copy-pasto while correcting a typo :) >> >>> It makes more >>> sense as we are materializing the tuple to write it. Similarly, in >>> the other change in the patch can use "write". >> >> Are you suggesting that we should use "write" in the modified comment >> instead of "inspect" in original comment. >> > > Yes. > > Another variant used in few other similar places is: > > /* > * get the heap tuple out of the tuple table slot, making sure we have a > * writable copy > */ > > >> Ok, I have now corrected grammar as well. Here's updated patch. >> > > Your new comment is also correct. I like the comment already used in > code as that makes the code consistent, what do you think? > I don't understand what do you mean by consistent. Do you mean to say that current usage " Store the slot into tuple ... " is correct? I don't think " so that we can write" is right usage either. I think, there should be a preposition after "write" something like "write to", to indicate that we will write to tuple. I find "scrible upon" to be better than "write to". But I am not wedded to any of those. However, I do want to change "store the slot into tuple" usage, which is wrong as explained earlier as well as in the commit message. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-26 13:32 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 15:13 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-27 04:39 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> @ 2018-06-27 05:54 ` Amit Kapila <[email protected]> 2018-06-27 06:00 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Amit Kapila @ 2018-06-27 05:54 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers On Wed, Jun 27, 2018 at 10:09 AM, Ashutosh Bapat <[email protected]> wrote: > On Tue, Jun 26, 2018 at 8:43 PM, Amit Kapila <[email protected]> wrote: >> On Tue, Jun 26, 2018 at 7:02 PM, Ashutosh Bapat >> <[email protected]> wrote: >>> On Tue, Jun 26, 2018 at 6:18 PM, Amit Kapila <[email protected]> wrote: >>>> On Tue, Jun 26, 2018 at 4:33 PM, Ashutosh Bapat >>>> <[email protected]> wrote: >>>>> Looks like we need similar adjustment in ExecSimpleRelationUpdate() as >>>>> well. Updated the patch. >>>>> >>>> >>>> - /* Store the slot into tuple that we can write. */ >>>> + /* Materialize slot into a tuple that we can inspect. */ >>>> tuple = ExecMaterializeSlot(slot); >>>> >>>> I think it is better to keep the last word of the sentence as "write" >>>> instead of "inspect" as was in the original sentence. >>> >>> A copy-pasto while correcting a typo :) >>> >>>> It makes more >>>> sense as we are materializing the tuple to write it. Similarly, in >>>> the other change in the patch can use "write". >>> >>> Are you suggesting that we should use "write" in the modified comment >>> instead of "inspect" in original comment. >>> >> >> Yes. >> >> Another variant used in few other similar places is: >> >> /* >> * get the heap tuple out of the tuple table slot, making sure we have a >> * writable copy >> */ >> >> >>> Ok, I have now corrected grammar as well. Here's updated patch. >>> >> >> Your new comment is also correct. I like the comment already used in >> code as that makes the code consistent, what do you think? >> > > I don't understand what do you mean by consistent. Do you mean to say > that current usage " Store the slot into tuple ... " is correct? > Oh no, I was talking about replacing it with below comment which is used at other places in the code. /* * get the heap tuple out of the tuple table slot, making sure we have a * writable copy */ -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-26 13:32 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 15:13 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-27 04:39 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-27 05:54 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> @ 2018-06-27 06:00 ` Ashutosh Bapat <[email protected]> 2018-06-27 13:15 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Ashutosh Bapat @ 2018-06-27 06:00 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: pgsql-hackers On Wed, Jun 27, 2018 at 11:24 AM, Amit Kapila <[email protected]> wrote: >> >> I don't understand what do you mean by consistent. Do you mean to say >> that current usage " Store the slot into tuple ... " is correct? >> > > Oh no, I was talking about replacing it with below comment which is > used at other places in the code. > /* > * get the heap tuple out of the tuple table slot, making sure we have a > * writable copy > */ Probably yes. But I would just correct the usage and grammar. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Thinko/typo in ExecSimpleRelationInsert 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-26 13:32 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 15:13 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-27 04:39 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-27 05:54 ` Re: Thinko/typo in ExecSimpleRelationInsert Amit Kapila <[email protected]> 2018-06-27 06:00 ` Re: Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> @ 2018-06-27 13:15 ` Amit Kapila <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Amit Kapila @ 2018-06-27 13:15 UTC (permalink / raw) To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers On Wed, Jun 27, 2018 at 11:30 AM, Ashutosh Bapat <[email protected]> wrote: > On Wed, Jun 27, 2018 at 11:24 AM, Amit Kapila <[email protected]> wrote: >>> >>> I don't understand what do you mean by consistent. Do you mean to say >>> that current usage " Store the slot into tuple ... " is correct? >>> >> >> Oh no, I was talking about replacing it with below comment which is >> used at other places in the code. >> /* >> * get the heap tuple out of the tuple table slot, making sure we have a >> * writable copy >> */ > > Probably yes. But I would just correct the usage and grammar. > Okay, pushed! -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 134 +++++--------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 39 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 182 insertions(+), 156 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index afc5ea5e0e7..20907ba5408 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * off_loc is the offset location required by the caller to use in error * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; /* * First, initialize the new pd_prune_xid value to zero (indicating no @@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + /* + * We can use frz_conflict_horizon as our cutoff for conflicts when + * the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. + */ + if (!(presult->all_visible_except_removable && presult->all_frozen)) + { + /* Avoids false conflicts when hot_standby_feedback in use */ + presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(presult->frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + presult->frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 74ebab25a95..c4553a4159c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.frz_conflict_horizon; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2339abfd28a..b2a4caeb33a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -210,9 +210,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -220,17 +221,18 @@ typedef struct PruneResult int8 htsv[MaxHeapTuplesPerPage + 1]; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 042d04c8de2..b2ddc1e2549 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2179,7 +2179,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --tez7m2a73jtztiij Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------ src/backend/access/heap/vacuumlazy.c | 129 ++++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 41 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 180 insertions(+), 151 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 44a5c0a917b..9c709315192 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,18 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/snapmgr.h" #include "utils/rel.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { Relation rel; @@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, - HeapPageFreeze *pagefrz, PruneResult *presult); + HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen, + PruneFreezeResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during * pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * off_loc is the offset location required by the caller to use in error * callback. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; /* * Our strategy is to scan the page and make lists of items to change, @@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer, if (pagefrz) prune_prepare_freeze_tuple(page, offnum, - pagefrz, presult); + pagefrz, frozen, presult); /* Ignore items already processed as part of an earlier chain */ if (prstate.marked[offnum]) @@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, HeapPageFreeze *pagefrz, - PruneResult *presult) + HeapTupleFreeze *frozen, + PruneFreezeResult *presult) { bool totally_frozen; HeapTupleHeader htup; @@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + frozen[presult->nfrozen++].offset = offnum; } /* @@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index abbb7ab3ada..6dd8d457c9c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel, static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static TransactionId heap_frz_conflict_horizon(PruneResult *presult, - HeapPageFreeze *pagefrz); - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. lazy_scan_prune must never become confused about whether a + * tuple should be frozen or removed. (In the future we might want to + * teach lazy_scan_prune to recompute vistest from time to time, to + * increase the number of dead tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * Determine the snapshotConflictHorizon for freezing. Must only be called * after pruning and determining if the page is freezable. */ -static TransactionId -heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) +TransactionId +heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz) { TransactionId result; @@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz); - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index 9eea1ed315a..7bffe09fb5d 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index bea35afc4bd..69d97bb8ece 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -204,9 +204,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -221,17 +222,18 @@ typedef struct PruneResult /* Number of newly frozen tuples */ int nfrozen; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + +extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult, + HeapPageFreeze *pagefrz); extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index aa7a25b8f8c..1c1a4d305d6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2175,7 +2175,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --rdqtp5puvxqotfdw Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------ src/backend/access/heap/vacuumlazy.c | 129 ++++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 41 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 180 insertions(+), 151 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 6bd8400b33b..abf6bdb2d99 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,18 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { Relation rel; @@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, - HeapPageFreeze *pagefrz, PruneResult *presult); + HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen, + PruneFreezeResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin) } /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during * pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * off_loc is the offset location required by the caller to use in error * callback. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; /* * Our strategy is to scan the page and make lists of items to change, @@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer, if (pagefrz) prune_prepare_freeze_tuple(page, offnum, - pagefrz, presult); + pagefrz, frozen, presult); /* Ignore items already processed as part of an earlier chain */ if (prstate.marked[offnum]) @@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, HeapPageFreeze *pagefrz, - PruneResult *presult) + HeapTupleFreeze *frozen, + PruneFreezeResult *presult) { bool totally_frozen; HeapTupleHeader htup; @@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + frozen[presult->nfrozen++].offset = offnum; } /* @@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index abbb7ab3ada..6dd8d457c9c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel, static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static TransactionId heap_frz_conflict_horizon(PruneResult *presult, - HeapPageFreeze *pagefrz); - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. lazy_scan_prune must never become confused about whether a + * tuple should be frozen or removed. (In the future we might want to + * teach lazy_scan_prune to recompute vistest from time to time, to + * increase the number of dead tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * Determine the snapshotConflictHorizon for freezing. Must only be called * after pruning and determining if the page is freezable. */ -static TransactionId -heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) +TransactionId +heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz) { TransactionId result; @@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz); - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2339abfd28a..45c4ae22e6a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -210,9 +210,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -220,17 +221,18 @@ typedef struct PruneResult int8 htsv[MaxHeapTuplesPerPage + 1]; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + +extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult, + HeapPageFreeze *pagefrz); extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index aa7a25b8f8c..1c1a4d305d6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2175,7 +2175,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --racicctn4wry6xe5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------ src/backend/access/heap/vacuumlazy.c | 129 ++++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 41 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 180 insertions(+), 151 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 44a5c0a917b..9c709315192 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,18 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/snapmgr.h" #include "utils/rel.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { Relation rel; @@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, - HeapPageFreeze *pagefrz, PruneResult *presult); + HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen, + PruneFreezeResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during * pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * off_loc is the offset location required by the caller to use in error * callback. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; /* * Our strategy is to scan the page and make lists of items to change, @@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer, if (pagefrz) prune_prepare_freeze_tuple(page, offnum, - pagefrz, presult); + pagefrz, frozen, presult); /* Ignore items already processed as part of an earlier chain */ if (prstate.marked[offnum]) @@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, HeapPageFreeze *pagefrz, - PruneResult *presult) + HeapTupleFreeze *frozen, + PruneFreezeResult *presult) { bool totally_frozen; HeapTupleHeader htup; @@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + frozen[presult->nfrozen++].offset = offnum; } /* @@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index abbb7ab3ada..6dd8d457c9c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel, static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static TransactionId heap_frz_conflict_horizon(PruneResult *presult, - HeapPageFreeze *pagefrz); - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. lazy_scan_prune must never become confused about whether a + * tuple should be frozen or removed. (In the future we might want to + * teach lazy_scan_prune to recompute vistest from time to time, to + * increase the number of dead tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * Determine the snapshotConflictHorizon for freezing. Must only be called * after pruning and determining if the page is freezable. */ -static TransactionId -heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) +TransactionId +heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz) { TransactionId result; @@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz); - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index 9eea1ed315a..7bffe09fb5d 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index bea35afc4bd..69d97bb8ece 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -204,9 +204,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -221,17 +222,18 @@ typedef struct PruneResult /* Number of newly frozen tuples */ int nfrozen; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + +extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult, + HeapPageFreeze *pagefrz); extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index aa7a25b8f8c..1c1a4d305d6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2175,7 +2175,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --rdqtp5puvxqotfdw Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------ src/backend/access/heap/vacuumlazy.c | 129 ++++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 41 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 180 insertions(+), 151 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 6bd8400b33b..abf6bdb2d99 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,18 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { Relation rel; @@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, - HeapPageFreeze *pagefrz, PruneResult *presult); + HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen, + PruneFreezeResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin) } /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during * pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * off_loc is the offset location required by the caller to use in error * callback. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; /* * Our strategy is to scan the page and make lists of items to change, @@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer, if (pagefrz) prune_prepare_freeze_tuple(page, offnum, - pagefrz, presult); + pagefrz, frozen, presult); /* Ignore items already processed as part of an earlier chain */ if (prstate.marked[offnum]) @@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, HeapPageFreeze *pagefrz, - PruneResult *presult) + HeapTupleFreeze *frozen, + PruneFreezeResult *presult) { bool totally_frozen; HeapTupleHeader htup; @@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + frozen[presult->nfrozen++].offset = offnum; } /* @@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index abbb7ab3ada..6dd8d457c9c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel, static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static TransactionId heap_frz_conflict_horizon(PruneResult *presult, - HeapPageFreeze *pagefrz); - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. lazy_scan_prune must never become confused about whether a + * tuple should be frozen or removed. (In the future we might want to + * teach lazy_scan_prune to recompute vistest from time to time, to + * increase the number of dead tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * Determine the snapshotConflictHorizon for freezing. Must only be called * after pruning and determining if the page is freezable. */ -static TransactionId -heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) +TransactionId +heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz) { TransactionId result; @@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz); - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2339abfd28a..45c4ae22e6a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -210,9 +210,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -220,17 +221,18 @@ typedef struct PruneResult int8 htsv[MaxHeapTuplesPerPage + 1]; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + +extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult, + HeapPageFreeze *pagefrz); extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index aa7a25b8f8c..1c1a4d305d6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2175,7 +2175,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --racicctn4wry6xe5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 134 +++++--------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 39 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 182 insertions(+), 156 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index afc5ea5e0e7..20907ba5408 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * off_loc is the offset location required by the caller to use in error * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; /* * First, initialize the new pd_prune_xid value to zero (indicating no @@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + /* + * We can use frz_conflict_horizon as our cutoff for conflicts when + * the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. + */ + if (!(presult->all_visible_except_removable && presult->all_frozen)) + { + /* Avoids false conflicts when hot_standby_feedback in use */ + presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(presult->frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + presult->frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 74ebab25a95..c4553a4159c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.frz_conflict_horizon; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2339abfd28a..b2a4caeb33a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -210,9 +210,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -220,17 +221,18 @@ typedef struct PruneResult int8 htsv[MaxHeapTuplesPerPage + 1]; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 042d04c8de2..b2ddc1e2549 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2179,7 +2179,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --tez7m2a73jtztiij Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------ src/backend/access/heap/vacuumlazy.c | 129 ++++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 41 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 180 insertions(+), 151 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 44a5c0a917b..9c709315192 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,18 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/snapmgr.h" #include "utils/rel.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { Relation rel; @@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, - HeapPageFreeze *pagefrz, PruneResult *presult); + HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen, + PruneFreezeResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during * pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * off_loc is the offset location required by the caller to use in error * callback. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; /* * Our strategy is to scan the page and make lists of items to change, @@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer, if (pagefrz) prune_prepare_freeze_tuple(page, offnum, - pagefrz, presult); + pagefrz, frozen, presult); /* Ignore items already processed as part of an earlier chain */ if (prstate.marked[offnum]) @@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, HeapPageFreeze *pagefrz, - PruneResult *presult) + HeapTupleFreeze *frozen, + PruneFreezeResult *presult) { bool totally_frozen; HeapTupleHeader htup; @@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + frozen[presult->nfrozen++].offset = offnum; } /* @@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index abbb7ab3ada..6dd8d457c9c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel, static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static TransactionId heap_frz_conflict_horizon(PruneResult *presult, - HeapPageFreeze *pagefrz); - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. lazy_scan_prune must never become confused about whether a + * tuple should be frozen or removed. (In the future we might want to + * teach lazy_scan_prune to recompute vistest from time to time, to + * increase the number of dead tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * Determine the snapshotConflictHorizon for freezing. Must only be called * after pruning and determining if the page is freezable. */ -static TransactionId -heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) +TransactionId +heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz) { TransactionId result; @@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz); - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index 9eea1ed315a..7bffe09fb5d 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index bea35afc4bd..69d97bb8ece 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -204,9 +204,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -221,17 +222,18 @@ typedef struct PruneResult /* Number of newly frozen tuples */ int nfrozen; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + +extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult, + HeapPageFreeze *pagefrz); extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index aa7a25b8f8c..1c1a4d305d6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2175,7 +2175,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --rdqtp5puvxqotfdw Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 134 +++++--------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 39 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 182 insertions(+), 156 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index afc5ea5e0e7..20907ba5408 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * off_loc is the offset location required by the caller to use in error * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; /* * First, initialize the new pd_prune_xid value to zero (indicating no @@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + /* + * We can use frz_conflict_horizon as our cutoff for conflicts when + * the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. + */ + if (!(presult->all_visible_except_removable && presult->all_frozen)) + { + /* Avoids false conflicts when hot_standby_feedback in use */ + presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(presult->frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + presult->frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 74ebab25a95..c4553a4159c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.frz_conflict_horizon; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2339abfd28a..b2a4caeb33a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -210,9 +210,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -220,17 +221,18 @@ typedef struct PruneResult int8 htsv[MaxHeapTuplesPerPage + 1]; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 042d04c8de2..b2ddc1e2549 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2179,7 +2179,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --tez7m2a73jtztiij Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune() @ 2024-03-08 21:45 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------ src/backend/access/heap/vacuumlazy.c | 129 ++++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 41 +++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 180 insertions(+), 151 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 680a50bf8b1..5e522f5b0ba 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 6bd8400b33b..abf6bdb2d99 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,18 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { Relation rel; @@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, - HeapPageFreeze *pagefrz, PruneResult *presult); + HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen, + PruneFreezeResult *presult); static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin) } /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED during * pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * off_loc is the offset location required by the caller to use in error * callback. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + bool do_freeze; + int64 fpi_before = pgWalUsage.wal_fpi; + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; /* * Our strategy is to scan the page and make lists of items to change, @@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer, /* for recovery conflicts */ presult->frz_conflict_horizon = InvalidTransactionId; + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; + maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(prstate.rel); @@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer, if (pagefrz) prune_prepare_freeze_tuple(page, offnum, - pagefrz, presult); + pagefrz, frozen, presult); /* Ignore items already processed as part of an earlier chain */ if (prstate.marked[offnum]) @@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (presult->all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* Caller won't update new_relfrozenxid and new_relminmxid */ + if (!pagefrz) + return; + + /* + * If we will freeze tuples on the page or, even if we don't freeze tuples + * on the page, if we will set the page all-frozen in the visibility map, + * we can advance relfrozenxid and relminmxid to the values in + * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } } @@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, HeapPageFreeze *pagefrz, - PruneResult *presult) + HeapTupleFreeze *frozen, + PruneFreezeResult *presult) { bool totally_frozen; HeapTupleHeader htup; @@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + frozen[presult->nfrozen++].offset = offnum; } /* @@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples @@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum) /* - * Perform the actual page changes needed by heap_page_prune. - * It is expected that the caller has a full cleanup lock on the - * buffer. + * Perform the actual page pruning modifications needed by + * heap_page_prune_and_freeze(). It is expected that the caller has a full + * cleanup lock on the buffer. */ void heap_page_prune_execute(Buffer buffer, @@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer, #ifdef USE_ASSERT_CHECKING /* - * When heap_page_prune() was called, mark_unused_now may have been - * passed as true, which allows would-be LP_DEAD items to be made - * LP_UNUSED instead. This is only possible if the relation has no - * indexes. If there are any dead items, then mark_unused_now was not - * true and every item being marked LP_UNUSED must refer to a + * When heap_page_prune_and_freeze() was called, mark_unused_now may + * have been passed as true, which allows would-be LP_DEAD items to be + * made LP_UNUSED instead. This is only possible if the relation has + * no indexes. If there are any dead items, then mark_unused_now was + * not true and every item being marked LP_UNUSED must refer to a * heap-only tuple. */ if (ndead > 0) diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index abbb7ab3ada..6dd8d457c9c 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel, static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static TransactionId heap_frz_conflict_horizon(PruneResult *presult, - HeapPageFreeze *pagefrz); - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. lazy_scan_prune must never become confused about whether a + * tuple should be frozen or removed. (In the future we might want to + * teach lazy_scan_prune to recompute vistest from time to time, to + * increase the number of dead tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * Determine the snapshotConflictHorizon for freezing. Must only be called * after pruning and determining if the page is freezable. */ -static TransactionId -heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) +TransactionId +heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz) { TransactionId result; @@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz) * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, - &pagefrz, &presult, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and check for tuples @@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz); - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) + if (presult.all_frozen) presult.frz_conflict_horizon = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 2339abfd28a..45c4ae22e6a 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,7 +195,7 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ @@ -210,9 +210,10 @@ typedef struct PruneResult /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -220,17 +221,18 @@ typedef struct PruneResult int8 htsv[MaxHeapTuplesPerPage + 1]; - /* - * One entry for every tuple that we may freeze. - */ - HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to - * guard against examining visibility status array members which have not yet - * been computed. + * of int8. See PruneFreezeResult.htsv for details. This helper function is + * meant to guard against examining visibility status array members which have + * not yet been computed. */ static inline HTSV_Result htsv_get_valid_status(int status) @@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + +extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult, + HeapPageFreeze *pagefrz); extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index aa7a25b8f8c..1c1a4d305d6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2175,7 +2175,7 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage -PruneResult +PruneFreezeResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --racicctn4wry6xe5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v7 08/16] Execute freezing in heap_page_prune() @ 2024-03-26 00:32 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 150 +++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 53 ++++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 225 insertions(+), 177 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2b7c7026429..4802d789963 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1048,7 +1048,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 457650ab651..e009c7579dd 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, PRUNE_ON_ACCESS, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, PRUNE_ON_ACCESS, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * reason indicates why the pruning is performed. It is included in the WAL * record for debugging and analysis purposes, but otherwise has no effect. @@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + TransactionId visibility_cutoff_xid; + bool do_freeze; + bool all_visible_except_removable; + int64 fpi_before = pgWalUsage.wal_fpi; /* * Our strategy is to scan the page and make lists of items to change, @@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer, * all_visible is also set to true. */ presult->all_frozen = true; - presult->all_visible = true; - /* for recovery conflicts */ - presult->visibility_cutoff_xid = InvalidTransactionId; + + /* + * The visibility cutoff xid is the newest xmin of live tuples on the + * page. In the common case, this will be set as the conflict horizon the + * caller can use for updating the VM. If, at the end of freezing and + * pruning, the page is all-frozen, there is no possibility that any + * running transaction on the standby does not see tuples on the page as + * all-visible, so the conflict horizon remains InvalidTransactionId. + */ + presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId; + + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer, * prefetching efficiency significantly / decreases the number of cache * misses. */ + all_visible_except_removable = true; for (offnum = maxoff; offnum >= FirstOffsetNumber; offnum = OffsetNumberPrev(offnum)) @@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer, * asynchronously. See SetHintBits for more info. Check that * the tuple is hinted xmin-committed because of that. */ - if (presult->all_visible) + if (all_visible_except_removable) { TransactionId xmin; if (!HeapTupleHeaderXminCommitted(htup)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } @@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer, if (xmin != FrozenTransactionId && !GlobalVisTestIsRemovableXid(vistest, xmin)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } /* Track newest xmin on page. */ - if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) && + if (TransactionIdFollows(xmin, visibility_cutoff_xid) && TransactionIdIsNormal(xmin)) - presult->visibility_cutoff_xid = xmin; + visibility_cutoff_xid = xmin; } break; case HEAPTUPLE_RECENTLY_DEAD: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_INSERT_IN_PROGRESS: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_DELETE_IN_PROGRESS: /* This is an expected case during concurrent vacuum */ - presult->all_visible = false; + all_visible_except_removable = false; break; default: elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result"); @@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer, * pruning and keep all_visible_except_removable to permit freezing if the * whole page will eventually become all visible after removing tuples. */ - presult->all_visible_except_removable = presult->all_visible; + presult->all_visible = all_visible_except_removable; /* Scan the page */ for (offnum = FirstOffsetNumber; @@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * We can use the visibility_cutoff_xid as our cutoff for conflicts + * when the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. This avoids false conflicts when + * hot_standby_feedback is in use. + */ + if (all_visible_except_removable && presult->all_frozen) + frz_conflict_horizon = visibility_cutoff_xid; + else + { + /* Avoids false conflicts when hot_standby_feedback in use */ + frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* + * For callers planning to update the visibility map, the conflict horizon + * for that record must be the newest xmin on the page. However, if the + * page is completely frozen, there can be no conflict and the + * vm_conflict_horizon should remain InvalidTransactionId. + */ + if (!presult->all_frozen) + presult->vm_conflict_horizon = visibility_cutoff_xid; + + if (pagefrz) + { + /* + * If we will freeze tuples on the page or, even if we don't freeze + * tuples on the page, if we will set the page all-frozen in the + * visibility map, we can advance relfrozenxid and relminmxid to the + * values in pagefrz->FreezePageRelfrozenXid and + * pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } + } } @@ -594,7 +705,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -859,10 +970,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -902,7 +1013,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -925,7 +1036,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index f474e661428..8beef4093ae 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz, - &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and update the variables set @@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.visibility_cutoff_xid; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) - presult.visibility_cutoff_xid = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* @@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel, Assert(presult.all_frozen == debug_all_frozen); Assert(!TransactionIdIsValid(debug_cutoff) || - debug_cutoff == presult.visibility_cutoff_xid); + debug_cutoff == presult.vm_conflict_horizon); } #endif @@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel, if (presult.all_frozen) { - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); flags |= VISIBILITYMAP_ALL_FROZEN; } @@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel, PageSetAllVisible(page); MarkBufferDirty(buf); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, presult.visibility_cutoff_xid, + vmbuffer, presult.vm_conflict_horizon, flags); } @@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel, /* * Set the page all-frozen (and all-visible) in the VM. * - * We can pass InvalidTransactionId as our visibility_cutoff_xid, - * since a snapshotConflictHorizon sufficient to make everything safe - * for REDO was logged when the page's tuples were frozen. + * We can pass InvalidTransactionId as our vm_conflict_horizon, since + * a snapshotConflictHorizon sufficient to make everything safe for + * REDO was logged when the page's tuples were frozen. */ - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, vmbuffer, InvalidTransactionId, VISIBILITYMAP_ALL_VISIBLE | diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 59c81f38e51..6f9c66a872b 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,13 +195,13 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ /* - * The rest of the fields in PruneResult are only guaranteed to be + * The rest of the fields in PruneFreezeResult are only guaranteed to be * initialized if heap_page_prune is passed PruneReason VACUUM_SCAN. */ @@ -212,23 +212,22 @@ typedef struct PruneResult */ bool all_visible; - /* - * Whether or not the page is all-visible except for tuples which will be - * removed during vacuum's second pass. This is used by VACUUM to - * determine whether or not to consider opportunistically freezing the - * page. - */ - bool all_visible_except_removable; - /* Whether or not the page can be set all-frozen in the VM */ bool all_frozen; - TransactionId visibility_cutoff_xid; /* Newest xmin on the page */ + + /* + * If the page is all-visible and not all-frozen this is the oldest xid + * that can see the page as all-visible. It is to be used as the snapshot + * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record. + */ + TransactionId vm_conflict_horizon; /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -242,9 +241,14 @@ typedef struct PruneResult * One entry for every tuple that we may freeze. */ HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; -/* 'reason' codes for heap_page_prune() */ + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; + +/* 'reason' codes for heap_page_prune_and_freeze() */ typedef enum { PRUNE_ON_ACCESS, /* on-access pruning */ @@ -254,7 +258,7 @@ typedef enum /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to + * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to * guard against examining visibility status array members which have not yet * been computed. */ @@ -332,6 +336,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -358,13 +363,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4679660837c..cc6a33ab3ee 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2192,8 +2192,8 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneFreezeResult PruneReason -PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --ck6erxojvlx23byk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v9 08/21] Execute freezing in heap_page_prune() @ 2024-03-26 00:32 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 150 +++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 52 ++++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 224 insertions(+), 177 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 6abfe36dec7..a793c0f56ee 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1106,7 +1106,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index eb09713311b..312695f806c 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, PRUNE_ON_ACCESS, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, PRUNE_ON_ACCESS, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * reason indicates why the pruning is performed. It is included in the WAL * record for debugging and analysis purposes, but otherwise has no effect. @@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + TransactionId visibility_cutoff_xid; + bool do_freeze; + bool all_visible_except_removable; + int64 fpi_before = pgWalUsage.wal_fpi; /* * Our strategy is to scan the page and make lists of items to change, @@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer, * all_visible is also set to true. */ presult->all_frozen = true; - presult->all_visible = true; - /* for recovery conflicts */ - presult->visibility_cutoff_xid = InvalidTransactionId; + + /* + * The visibility cutoff xid is the newest xmin of live tuples on the + * page. In the common case, this will be set as the conflict horizon the + * caller can use for updating the VM. If, at the end of freezing and + * pruning, the page is all-frozen, there is no possibility that any + * running transaction on the standby does not see tuples on the page as + * all-visible, so the conflict horizon remains InvalidTransactionId. + */ + presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId; + + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer, * prefetching efficiency significantly / decreases the number of cache * misses. */ + all_visible_except_removable = true; for (offnum = maxoff; offnum >= FirstOffsetNumber; offnum = OffsetNumberPrev(offnum)) @@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer, * asynchronously. See SetHintBits for more info. Check that * the tuple is hinted xmin-committed because of that. */ - if (presult->all_visible) + if (all_visible_except_removable) { TransactionId xmin; if (!HeapTupleHeaderXminCommitted(htup)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } @@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer, if (xmin != FrozenTransactionId && !GlobalVisTestIsRemovableXid(vistest, xmin)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } /* Track newest xmin on page. */ - if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) && + if (TransactionIdFollows(xmin, visibility_cutoff_xid) && TransactionIdIsNormal(xmin)) - presult->visibility_cutoff_xid = xmin; + visibility_cutoff_xid = xmin; } break; case HEAPTUPLE_RECENTLY_DEAD: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_INSERT_IN_PROGRESS: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_DELETE_IN_PROGRESS: /* This is an expected case during concurrent vacuum */ - presult->all_visible = false; + all_visible_except_removable = false; break; default: elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result"); @@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer, * pruning and keep all_visible_except_removable to permit freezing if the * whole page will eventually become all visible after removing tuples. */ - presult->all_visible_except_removable = presult->all_visible; + presult->all_visible = all_visible_except_removable; /* Scan the page */ for (offnum = FirstOffsetNumber; @@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * We can use the visibility_cutoff_xid as our cutoff for conflicts + * when the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. This avoids false conflicts when + * hot_standby_feedback is in use. + */ + if (all_visible_except_removable && presult->all_frozen) + frz_conflict_horizon = visibility_cutoff_xid; + else + { + /* Avoids false conflicts when hot_standby_feedback in use */ + frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* + * For callers planning to update the visibility map, the conflict horizon + * for that record must be the newest xmin on the page. However, if the + * page is completely frozen, there can be no conflict and the + * vm_conflict_horizon should remain InvalidTransactionId. + */ + if (!presult->all_frozen) + presult->vm_conflict_horizon = visibility_cutoff_xid; + + if (pagefrz) + { + /* + * If we will freeze tuples on the page or, even if we don't freeze + * tuples on the page, if we will set the page all-frozen in the + * visibility map, we can advance relfrozenxid and relminmxid to the + * values in pagefrz->FreezePageRelfrozenXid and + * pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } + } } @@ -590,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -855,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -898,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -921,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index f474e661428..8beef4093ae 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz, - &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and update the variables set @@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.visibility_cutoff_xid; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) - presult.visibility_cutoff_xid = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* @@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel, Assert(presult.all_frozen == debug_all_frozen); Assert(!TransactionIdIsValid(debug_cutoff) || - debug_cutoff == presult.visibility_cutoff_xid); + debug_cutoff == presult.vm_conflict_horizon); } #endif @@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel, if (presult.all_frozen) { - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); flags |= VISIBILITYMAP_ALL_FROZEN; } @@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel, PageSetAllVisible(page); MarkBufferDirty(buf); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, presult.visibility_cutoff_xid, + vmbuffer, presult.vm_conflict_horizon, flags); } @@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel, /* * Set the page all-frozen (and all-visible) in the VM. * - * We can pass InvalidTransactionId as our visibility_cutoff_xid, - * since a snapshotConflictHorizon sufficient to make everything safe - * for REDO was logged when the page's tuples were frozen. + * We can pass InvalidTransactionId as our vm_conflict_horizon, since + * a snapshotConflictHorizon sufficient to make everything safe for + * REDO was logged when the page's tuples were frozen. */ - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, vmbuffer, InvalidTransactionId, VISIBILITYMAP_ALL_VISIBLE | diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 9d047621ea5..de11c166575 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,13 +195,13 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ /* - * The rest of the fields in PruneResult are only guaranteed to be + * The rest of the fields in PruneFreezeResult are only guaranteed to be * initialized if heap_page_prune is passed PruneReason VACUUM_SCAN. */ @@ -212,23 +212,22 @@ typedef struct PruneResult */ bool all_visible; - /* - * Whether or not the page is all-visible except for tuples which will be - * removed during vacuum's second pass. This is used by VACUUM to - * determine whether or not to consider opportunistically freezing the - * page. - */ - bool all_visible_except_removable; - /* Whether or not the page can be set all-frozen in the VM */ bool all_frozen; - TransactionId visibility_cutoff_xid; /* Newest xmin on the page */ + + /* + * If the page is all-visible and not all-frozen this is the oldest xid + * that can see the page as all-visible. It is to be used as the snapshot + * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record. + */ + TransactionId vm_conflict_horizon; /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -242,9 +241,14 @@ typedef struct PruneResult * One entry for every tuple that we may freeze. */ HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; -/* 'reason' codes for heap_page_prune() */ +/* 'reason' codes for heap_page_prune_and_freeze() */ typedef enum { PRUNE_ON_ACCESS, /* on-access pruning */ @@ -254,7 +258,7 @@ typedef enum /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to + * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to * guard against examining visibility status array members which have not yet * been computed. */ @@ -361,13 +365,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index cfa9d5aaeac..5737bc5b945 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2191,8 +2191,8 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneFreezeResult PruneReason -PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --caj67xgx3lukmr5f Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v9 08/21] Execute freezing in heap_page_prune() @ 2024-03-26 00:32 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 150 +++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 52 ++++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 224 insertions(+), 177 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 6abfe36dec7..a793c0f56ee 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1106,7 +1106,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index eb09713311b..312695f806c 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, PRUNE_ON_ACCESS, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, PRUNE_ON_ACCESS, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * reason indicates why the pruning is performed. It is included in the WAL * record for debugging and analysis purposes, but otherwise has no effect. @@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + TransactionId visibility_cutoff_xid; + bool do_freeze; + bool all_visible_except_removable; + int64 fpi_before = pgWalUsage.wal_fpi; /* * Our strategy is to scan the page and make lists of items to change, @@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer, * all_visible is also set to true. */ presult->all_frozen = true; - presult->all_visible = true; - /* for recovery conflicts */ - presult->visibility_cutoff_xid = InvalidTransactionId; + + /* + * The visibility cutoff xid is the newest xmin of live tuples on the + * page. In the common case, this will be set as the conflict horizon the + * caller can use for updating the VM. If, at the end of freezing and + * pruning, the page is all-frozen, there is no possibility that any + * running transaction on the standby does not see tuples on the page as + * all-visible, so the conflict horizon remains InvalidTransactionId. + */ + presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId; + + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer, * prefetching efficiency significantly / decreases the number of cache * misses. */ + all_visible_except_removable = true; for (offnum = maxoff; offnum >= FirstOffsetNumber; offnum = OffsetNumberPrev(offnum)) @@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer, * asynchronously. See SetHintBits for more info. Check that * the tuple is hinted xmin-committed because of that. */ - if (presult->all_visible) + if (all_visible_except_removable) { TransactionId xmin; if (!HeapTupleHeaderXminCommitted(htup)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } @@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer, if (xmin != FrozenTransactionId && !GlobalVisTestIsRemovableXid(vistest, xmin)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } /* Track newest xmin on page. */ - if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) && + if (TransactionIdFollows(xmin, visibility_cutoff_xid) && TransactionIdIsNormal(xmin)) - presult->visibility_cutoff_xid = xmin; + visibility_cutoff_xid = xmin; } break; case HEAPTUPLE_RECENTLY_DEAD: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_INSERT_IN_PROGRESS: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_DELETE_IN_PROGRESS: /* This is an expected case during concurrent vacuum */ - presult->all_visible = false; + all_visible_except_removable = false; break; default: elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result"); @@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer, * pruning and keep all_visible_except_removable to permit freezing if the * whole page will eventually become all visible after removing tuples. */ - presult->all_visible_except_removable = presult->all_visible; + presult->all_visible = all_visible_except_removable; /* Scan the page */ for (offnum = FirstOffsetNumber; @@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * We can use the visibility_cutoff_xid as our cutoff for conflicts + * when the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. This avoids false conflicts when + * hot_standby_feedback is in use. + */ + if (all_visible_except_removable && presult->all_frozen) + frz_conflict_horizon = visibility_cutoff_xid; + else + { + /* Avoids false conflicts when hot_standby_feedback in use */ + frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* + * For callers planning to update the visibility map, the conflict horizon + * for that record must be the newest xmin on the page. However, if the + * page is completely frozen, there can be no conflict and the + * vm_conflict_horizon should remain InvalidTransactionId. + */ + if (!presult->all_frozen) + presult->vm_conflict_horizon = visibility_cutoff_xid; + + if (pagefrz) + { + /* + * If we will freeze tuples on the page or, even if we don't freeze + * tuples on the page, if we will set the page all-frozen in the + * visibility map, we can advance relfrozenxid and relminmxid to the + * values in pagefrz->FreezePageRelfrozenXid and + * pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } + } } @@ -590,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -855,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -898,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -921,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index f474e661428..8beef4093ae 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz, - &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and update the variables set @@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.visibility_cutoff_xid; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) - presult.visibility_cutoff_xid = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* @@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel, Assert(presult.all_frozen == debug_all_frozen); Assert(!TransactionIdIsValid(debug_cutoff) || - debug_cutoff == presult.visibility_cutoff_xid); + debug_cutoff == presult.vm_conflict_horizon); } #endif @@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel, if (presult.all_frozen) { - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); flags |= VISIBILITYMAP_ALL_FROZEN; } @@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel, PageSetAllVisible(page); MarkBufferDirty(buf); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, presult.visibility_cutoff_xid, + vmbuffer, presult.vm_conflict_horizon, flags); } @@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel, /* * Set the page all-frozen (and all-visible) in the VM. * - * We can pass InvalidTransactionId as our visibility_cutoff_xid, - * since a snapshotConflictHorizon sufficient to make everything safe - * for REDO was logged when the page's tuples were frozen. + * We can pass InvalidTransactionId as our vm_conflict_horizon, since + * a snapshotConflictHorizon sufficient to make everything safe for + * REDO was logged when the page's tuples were frozen. */ - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, vmbuffer, InvalidTransactionId, VISIBILITYMAP_ALL_VISIBLE | diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 9d047621ea5..de11c166575 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,13 +195,13 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ /* - * The rest of the fields in PruneResult are only guaranteed to be + * The rest of the fields in PruneFreezeResult are only guaranteed to be * initialized if heap_page_prune is passed PruneReason VACUUM_SCAN. */ @@ -212,23 +212,22 @@ typedef struct PruneResult */ bool all_visible; - /* - * Whether or not the page is all-visible except for tuples which will be - * removed during vacuum's second pass. This is used by VACUUM to - * determine whether or not to consider opportunistically freezing the - * page. - */ - bool all_visible_except_removable; - /* Whether or not the page can be set all-frozen in the VM */ bool all_frozen; - TransactionId visibility_cutoff_xid; /* Newest xmin on the page */ + + /* + * If the page is all-visible and not all-frozen this is the oldest xid + * that can see the page as all-visible. It is to be used as the snapshot + * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record. + */ + TransactionId vm_conflict_horizon; /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -242,9 +241,14 @@ typedef struct PruneResult * One entry for every tuple that we may freeze. */ HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; + + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; -/* 'reason' codes for heap_page_prune() */ +/* 'reason' codes for heap_page_prune_and_freeze() */ typedef enum { PRUNE_ON_ACCESS, /* on-access pruning */ @@ -254,7 +258,7 @@ typedef enum /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to + * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to * guard against examining visibility status array members which have not yet * been computed. */ @@ -361,13 +365,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index cfa9d5aaeac..5737bc5b945 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2191,8 +2191,8 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneFreezeResult PruneReason -PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --caj67xgx3lukmr5f Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v7 08/16] Execute freezing in heap_page_prune() @ 2024-03-26 00:32 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw) As a step toward combining the prune and freeze WAL records, execute freezing in heap_page_prune(). The logic to determine whether or not to execute freeze plans was moved from lazy_scan_prune() over to heap_page_prune() with little modification. --- src/backend/access/heap/heapam_handler.c | 2 +- src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++----- src/backend/access/heap/vacuumlazy.c | 150 +++++------------- src/backend/storage/ipc/procarray.c | 6 +- src/include/access/heapam.h | 53 ++++--- src/tools/pgindent/typedefs.list | 2 +- 6 files changed, 225 insertions(+), 177 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2b7c7026429..4802d789963 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1048,7 +1048,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * We ignore unused and redirect line pointers. DEAD line pointers * should be counted as dead, because we need vacuum to run to get rid * of them. Note that this rule agrees with the way that - * heap_page_prune() counts things. + * heap_page_prune_and_freeze() counts things. */ if (!ItemIdIsNormal(itemid)) { diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 457650ab651..e009c7579dd 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -17,16 +17,19 @@ #include "access/heapam.h" #include "access/heapam_xlog.h" #include "access/htup_details.h" +#include "access/multixact.h" #include "access/transam.h" #include "access/xlog.h" +#include "commands/vacuum.h" #include "access/xloginsert.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/bufmgr.h" #include "utils/rel.h" #include "utils/snapmgr.h" -/* Working data for heap_page_prune and subroutines */ +/* Working data for heap_page_prune_and_freeze() and subroutines */ typedef struct { /* tuple visibility test, initialized for the relation */ @@ -51,6 +54,11 @@ typedef struct * 1. Otherwise every access would need to subtract 1. */ bool marked[MaxHeapTuplesPerPage + 1]; + + /* + * One entry for every tuple that we may freeze. + */ + HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; } PruneState; /* Local functions */ @@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate, Buffer buffer); static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult); + PruneState *prstate, PruneFreezeResult *presult); + static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid); static void heap_prune_record_redirect(PruneState *prstate, OffsetNumber offnum, OffsetNumber rdoffnum); static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult); + PruneFreezeResult *presult); static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum); static void page_verify_redirects(Page page); @@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer) */ if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree) { - PruneResult presult; + PruneFreezeResult presult; /* * For now, pass mark_unused_now as false regardless of whether or * not the relation has indexes, since we cannot safely determine * that during on-access pruning with the current implementation. */ - heap_page_prune(relation, buffer, vistest, false, NULL, - &presult, PRUNE_ON_ACCESS, NULL); + heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL, + &presult, PRUNE_ON_ACCESS, NULL); /* * Report the number of tuples reclaimed to pgstats. This is @@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer) /* - * Prune and repair fragmentation in the specified page. + * Prune and repair fragmentation and potentially freeze tuples on the + * specified page. + * + * If the page can be marked all-frozen in the visibility map, we may + * opportunistically freeze tuples on the page if either its tuples are old + * enough or freezing will be cheap enough. * * Caller must have pin and buffer cleanup lock on the page. Note that we * don't update the FSM information for page on caller's behalf. Caller might @@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * mark_unused_now indicates whether or not dead items can be set LP_UNUSED * during pruning. * - * pagefrz contains both input and output parameters used if the caller is - * interested in potentially freezing tuples on the page. + * pagefrz is an input parameter containing visibility cutoff information and + * the current relfrozenxid and relminmxids used if the caller is interested in + * freezing tuples on the page. * * presult contains output parameters needed by callers such as the number of * tuples removed and the number of line pointers newly marked LP_DEAD. - * heap_page_prune() is responsible for initializing it. + * heap_page_prune_and_freeze() is responsible for initializing it. * * reason indicates why the pruning is performed. It is included in the WAL * record for debugging and analysis purposes, but otherwise has no effect. @@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer) * callback. */ void -heap_page_prune(Relation relation, Buffer buffer, - GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc) +heap_page_prune_and_freeze(Relation relation, Buffer buffer, + GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc) { Page page = BufferGetPage(buffer); BlockNumber blockno = BufferGetBlockNumber(buffer); @@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer, maxoff; PruneState prstate; HeapTupleData tup; + TransactionId visibility_cutoff_xid; + bool do_freeze; + bool all_visible_except_removable; + int64 fpi_before = pgWalUsage.wal_fpi; /* * Our strategy is to scan the page and make lists of items to change, @@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer, * all_visible is also set to true. */ presult->all_frozen = true; - presult->all_visible = true; - /* for recovery conflicts */ - presult->visibility_cutoff_xid = InvalidTransactionId; + + /* + * The visibility cutoff xid is the newest xmin of live tuples on the + * page. In the common case, this will be set as the conflict horizon the + * caller can use for updating the VM. If, at the end of freezing and + * pruning, the page is all-frozen, there is no possibility that any + * running transaction on the standby does not see tuples on the page as + * all-visible, so the conflict horizon remains InvalidTransactionId. + */ + presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId; + + /* For advancing relfrozenxid and relminmxid */ + presult->new_relfrozenxid = InvalidTransactionId; + presult->new_relminmxid = InvalidMultiXactId; maxoff = PageGetMaxOffsetNumber(page); tup.t_tableOid = RelationGetRelid(relation); @@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer, * prefetching efficiency significantly / decreases the number of cache * misses. */ + all_visible_except_removable = true; for (offnum = maxoff; offnum >= FirstOffsetNumber; offnum = OffsetNumberPrev(offnum)) @@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer, * asynchronously. See SetHintBits for more info. Check that * the tuple is hinted xmin-committed because of that. */ - if (presult->all_visible) + if (all_visible_except_removable) { TransactionId xmin; if (!HeapTupleHeaderXminCommitted(htup)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } @@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer, if (xmin != FrozenTransactionId && !GlobalVisTestIsRemovableXid(vistest, xmin)) { - presult->all_visible = false; + all_visible_except_removable = false; break; } /* Track newest xmin on page. */ - if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) && + if (TransactionIdFollows(xmin, visibility_cutoff_xid) && TransactionIdIsNormal(xmin)) - presult->visibility_cutoff_xid = xmin; + visibility_cutoff_xid = xmin; } break; case HEAPTUPLE_RECENTLY_DEAD: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_INSERT_IN_PROGRESS: - presult->all_visible = false; + all_visible_except_removable = false; break; case HEAPTUPLE_DELETE_IN_PROGRESS: /* This is an expected case during concurrent vacuum */ - presult->all_visible = false; + all_visible_except_removable = false; break; default: elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result"); @@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer, /* Tuple with storage -- consider need to freeze */ if ((heap_prepare_freeze_tuple(htup, pagefrz, - &presult->frozen[presult->nfrozen], + &prstate.frozen[presult->nfrozen], &totally_frozen))) { /* Save prepared freeze plan for later */ - presult->frozen[presult->nfrozen++].offset = offnum; + prstate.frozen[presult->nfrozen++].offset = offnum; } /* @@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer, * pruning and keep all_visible_except_removable to permit freezing if the * whole page will eventually become all visible after removing tuples. */ - presult->all_visible_except_removable = presult->all_visible; + presult->all_visible = all_visible_except_removable; /* Scan the page */ for (offnum = FirstOffsetNumber; @@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + */ + if (pagefrz) + do_freeze = pagefrz->freeze_required || + (all_visible_except_removable && presult->all_frozen && + presult->nfrozen > 0 && + fpi_before != pgWalUsage.wal_fpi); + else + do_freeze = false; + + if (do_freeze) + { + TransactionId frz_conflict_horizon = InvalidTransactionId; + + /* + * We can use the visibility_cutoff_xid as our cutoff for conflicts + * when the whole page is eligible to become all-frozen in the VM once + * we're done with it. Otherwise we generate a conservative cutoff by + * stepping back from OldestXmin. This avoids false conflicts when + * hot_standby_feedback is in use. + */ + if (all_visible_except_removable && presult->all_frozen) + frz_conflict_horizon = visibility_cutoff_xid; + else + { + /* Avoids false conflicts when hot_standby_feedback in use */ + frz_conflict_horizon = pagefrz->cutoffs->OldestXmin; + TransactionIdRetreat(frz_conflict_horizon); + } + + /* Execute all freeze plans for page as a single atomic action */ + heap_freeze_execute_prepared(relation, buffer, + frz_conflict_horizon, + prstate.frozen, presult->nfrozen); + } + else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0) + { + /* + * If we will neither freeze tuples on the page nor set the page all + * frozen in the visibility map, the page is not all frozen and there + * will be no newly frozen tuples. + */ + presult->all_frozen = false; + presult->nfrozen = 0; /* avoid miscounts in instrumentation */ + } + + /* + * For callers planning to update the visibility map, the conflict horizon + * for that record must be the newest xmin on the page. However, if the + * page is completely frozen, there can be no conflict and the + * vm_conflict_horizon should remain InvalidTransactionId. + */ + if (!presult->all_frozen) + presult->vm_conflict_horizon = visibility_cutoff_xid; + + if (pagefrz) + { + /* + * If we will freeze tuples on the page or, even if we don't freeze + * tuples on the page, if we will set the page all-frozen in the + * visibility map, we can advance relfrozenxid and relminmxid to the + * values in pagefrz->FreezePageRelfrozenXid and + * pagefrz->FreezePageRelminMxid. + */ + if (presult->all_frozen || presult->nfrozen > 0) + { + presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->FreezePageRelminMxid; + } + else + { + presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid; + presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid; + } + } } @@ -594,7 +705,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer) */ static int heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, - PruneState *prstate, PruneResult *presult) + PruneState *prstate, PruneFreezeResult *presult) { int ndeleted = 0; Page dp = (Page) BufferGetPage(buffer); @@ -859,10 +970,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, { /* * We found a redirect item that doesn't point to a valid follow-on - * item. This can happen if the loop in heap_page_prune caused us to - * visit the dead successor of a redirect item before visiting the - * redirect item. We can clean up by setting the redirect item to - * DEAD state or LP_UNUSED if the caller indicated. + * item. This can happen if the loop in heap_page_prune_and_freeze() + * caused us to visit the dead successor of a redirect item before + * visiting the redirect item. We can clean up by setting the + * redirect item to DEAD state or LP_UNUSED if the caller indicated. */ heap_prune_record_dead_or_unused(prstate, rootoffnum, presult); } @@ -902,7 +1013,7 @@ heap_prune_record_redirect(PruneState *prstate, /* Record line pointer to be marked dead */ static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { Assert(prstate->ndead < MaxHeapTuplesPerPage); prstate->nowdead[prstate->ndead] = offnum; @@ -925,7 +1036,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum, */ static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum, - PruneResult *presult) + PruneFreezeResult *presult) { /* * If the caller set mark_unused_now to true, we can remove dead tuples diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index f474e661428..8beef4093ae 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, * as an upper bound on the XIDs stored in the pages we'll actually scan * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs). * - * Next acquire vistest, a related cutoff that's used in heap_page_prune. - * We expect vistest will always make heap_page_prune remove any deleted - * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become - * confused about whether a tuple should be frozen or removed. (In the - * future we might want to teach lazy_scan_prune to recompute vistest from - * time to time, to increase the number of dead tuples it can prune away.) + * Next acquire vistest, a related cutoff that's used in + * heap_page_prune_and_freeze(). We expect vistest will always make + * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is < + * OldestXmin. (In the future we might want to teach lazy_scan_prune to + * recompute vistest from time to time, to increase the number of dead + * tuples it can prune away.) */ vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); @@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno, * * Caller must hold pin and buffer cleanup lock on the buffer. * - * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune() - * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about - * whether or not a tuple should be considered DEAD. This happened when an - * inserting transaction concurrently aborted (after our heap_page_prune() - * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot - * of complexity just so we could deal with tuples that were DEAD to VACUUM, - * but nevertheless were left with storage after pruning. + * Prior to PostgreSQL 14 there were very rare cases where + * heap_page_prune_and_freeze() was allowed to disagree with our + * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be + * considered DEAD. This happened when an inserting transaction concurrently + * aborted (after our heap_page_prune_and_freeze() call, before our + * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just + * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were + * left with storage after pruning. * * As of Postgres 17, we circumvent this problem altogether by reusing the - * result of heap_page_prune()'s visibility check. Without the second call to - * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no - * disagreement. We'll just handle such tuples as if they had become fully dead - * right after this operation completes instead of in the middle of it. Note that - * any tuple that becomes dead after the call to heap_page_prune() can't need to - * be frozen, because it was visible to another session when vacuum started. + * result of heap_page_prune_and_freeze()'s visibility check. Without the + * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and + * there can be no disagreement. We'll just handle such tuples as if they had + * become fully dead right after this operation completes instead of in the + * middle of it. * * vmbuffer is the buffer containing the VM block with visibility information * for the heap block, blkno. all_visible_according_to_vm is the saved @@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel, OffsetNumber offnum, maxoff; ItemId itemid; - PruneResult presult; + PruneFreezeResult presult; int lpdead_items, live_tuples, recently_dead_tuples; HeapPageFreeze pagefrz; bool hastup = false; - bool do_freeze; - int64 fpi_before = pgWalUsage.wal_fpi; OffsetNumber deadoffsets[MaxHeapTuplesPerPage]; Assert(BufferGetBlockNumber(buf) == blkno); /* * maxoff might be reduced following line pointer array truncation in - * heap_page_prune. That's safe for us to ignore, since the reclaimed - * space will continue to look like LP_UNUSED items below. + * heap_page_prune_and_freeze(). That's safe for us to ignore, since the + * reclaimed space will continue to look like LP_UNUSED items below. */ maxoff = PageGetMaxOffsetNumber(page); - /* Initialize (or reset) page-level state */ + /* Initialize pagefrz */ pagefrz.freeze_required = false; pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid; pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid; @@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel, recently_dead_tuples = 0; /* - * Prune all HOT-update chains in this page. + * Prune all HOT-update chains and potentially freeze tuples on this page. * * We count the number of tuples removed from the page by the pruning step * in presult.ndeleted. It should not be confused with lpdead_items; @@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel, * items LP_UNUSED, so mark_unused_now should be true if no indexes and * false otherwise. */ - heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz, - &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); + heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0, + &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum); /* * Now scan the page to collect LP_DEAD items and update the variables set @@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel, vacrel->offnum = InvalidOffsetNumber; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - do_freeze = pagefrz.freeze_required || - (presult.all_visible_except_removable && presult.all_frozen && - presult.nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); + Assert(MultiXactIdIsValid(presult.new_relminmxid)); + vacrel->NewRelfrozenXid = presult.new_relfrozenxid; + Assert(TransactionIdIsValid(presult.new_relfrozenxid)); + vacrel->NewRelminMxid = presult.new_relminmxid; - if (do_freeze) + if (presult.nfrozen > 0) { - TransactionId snapshotConflictHorizon; - /* - * We're freezing the page. Our final NewRelfrozenXid doesn't need to - * be affected by the XIDs that are just about to be frozen anyway. + * We never increment the frozen_pages instrumentation counter when + * nfrozen == 0, since it only counts pages with newly frozen tuples + * (don't confuse that with pages newly set all-frozen in VM). */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - vacrel->frozen_pages++; - /* - * We can use frz_conflict_horizon as our cutoff for conflicts when - * the whole page is eligible to become all-frozen in the VM once - * we're done with it. Otherwise we generate a conservative cutoff by - * stepping back from OldestXmin. - */ - if (presult.all_visible_except_removable && presult.all_frozen) - snapshotConflictHorizon = presult.visibility_cutoff_xid; - else - { - /* Avoids false conflicts when hot_standby_feedback in use */ - snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin; - TransactionIdRetreat(snapshotConflictHorizon); - } - - /* Using same cutoff when setting VM is now unnecessary */ - if (presult.all_visible_except_removable && presult.all_frozen) - presult.visibility_cutoff_xid = InvalidTransactionId; - - /* Execute all freeze plans for page as a single atomic action */ - heap_freeze_execute_prepared(vacrel->rel, buf, - snapshotConflictHorizon, - presult.frozen, presult.nfrozen); - - } - else if (presult.all_frozen && presult.nfrozen == 0) - { - /* Page should be all visible except to-be-removed tuples */ - Assert(presult.all_visible_except_removable); - - /* - * We have no freeze plans to execute, so there's no added cost from - * following the freeze path. That's why it was chosen. This is - * important in the case where the page only contains totally frozen - * tuples at this point (perhaps only following pruning). Such pages - * can be marked all-frozen in the VM by our caller, even though none - * of its tuples were newly frozen here (note that the "no freeze" - * path never sets pages all-frozen). - * - * We never increment the frozen_pages instrumentation counter here, - * since it only counts pages with newly frozen tuples (don't confuse - * that with pages newly set all-frozen in VM). - */ - vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid; - } - else - { - /* - * Page requires "no freeze" processing. It might be set all-visible - * in the visibility map, but it can never be set all-frozen. - */ - vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid; - vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid; - presult.all_frozen = false; - presult.nfrozen = 0; /* avoid miscounts in instrumentation */ } /* @@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel, Assert(presult.all_frozen == debug_all_frozen); Assert(!TransactionIdIsValid(debug_cutoff) || - debug_cutoff == presult.visibility_cutoff_xid); + debug_cutoff == presult.vm_conflict_horizon); } #endif @@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel, if (presult.all_frozen) { - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); flags |= VISIBILITYMAP_ALL_FROZEN; } @@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel, PageSetAllVisible(page); MarkBufferDirty(buf); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, presult.visibility_cutoff_xid, + vmbuffer, presult.vm_conflict_horizon, flags); } @@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel, /* * Set the page all-frozen (and all-visible) in the VM. * - * We can pass InvalidTransactionId as our visibility_cutoff_xid, - * since a snapshotConflictHorizon sufficient to make everything safe - * for REDO was logged when the page's tuples were frozen. + * We can pass InvalidTransactionId as our vm_conflict_horizon, since + * a snapshotConflictHorizon sufficient to make everything safe for + * REDO was logged when the page's tuples were frozen. */ - Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid)); + Assert(!TransactionIdIsValid(presult.vm_conflict_horizon)); visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr, vmbuffer, InvalidTransactionId, VISIBILITYMAP_ALL_VISIBLE | diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b3cd248fb64..88a6d504dff 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid) * Note: the approximate horizons (see definition of GlobalVisState) are * updated by the computations done here. That's currently required for * correctness and a small optimization. Without doing so it's possible that - * heap vacuum's call to heap_page_prune() uses a more conservative horizon - * than later when deciding which tuples can be removed - which the code - * doesn't expect (breaking HOT). + * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative + * horizon than later when deciding which tuples can be removed - which the + * code doesn't expect (breaking HOT). */ static void ComputeXidHorizons(ComputeXidHorizonsResult *h) diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 59c81f38e51..6f9c66a872b 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -195,13 +195,13 @@ typedef struct HeapPageFreeze /* * Per-page state returned from pruning */ -typedef struct PruneResult +typedef struct PruneFreezeResult { int ndeleted; /* Number of tuples deleted from the page */ int nnewlpdead; /* Number of newly LP_DEAD items */ /* - * The rest of the fields in PruneResult are only guaranteed to be + * The rest of the fields in PruneFreezeResult are only guaranteed to be * initialized if heap_page_prune is passed PruneReason VACUUM_SCAN. */ @@ -212,23 +212,22 @@ typedef struct PruneResult */ bool all_visible; - /* - * Whether or not the page is all-visible except for tuples which will be - * removed during vacuum's second pass. This is used by VACUUM to - * determine whether or not to consider opportunistically freezing the - * page. - */ - bool all_visible_except_removable; - /* Whether or not the page can be set all-frozen in the VM */ bool all_frozen; - TransactionId visibility_cutoff_xid; /* Newest xmin on the page */ + + /* + * If the page is all-visible and not all-frozen this is the oldest xid + * that can see the page as all-visible. It is to be used as the snapshot + * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record. + */ + TransactionId vm_conflict_horizon; /* * Tuple visibility is only computed once for each tuple, for correctness - * and efficiency reasons; see comment in heap_page_prune() for details. - * This is of type int8[], instead of HTSV_Result[], so we can use -1 to - * indicate no visibility has been computed, e.g. for LP_DEAD items. + * and efficiency reasons; see comment in heap_page_prune_and_freeze() for + * details. This is of type int8[], instead of HTSV_Result[], so we can + * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD + * items. * * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is * 1. Otherwise every access would need to subtract 1. @@ -242,9 +241,14 @@ typedef struct PruneResult * One entry for every tuple that we may freeze. */ HeapTupleFreeze frozen[MaxHeapTuplesPerPage]; -} PruneResult; + /* New value of relfrozenxid found by heap_page_prune_and_freeze() */ + TransactionId new_relfrozenxid; -/* 'reason' codes for heap_page_prune() */ + /* New value of relminmxid found by heap_page_prune_and_freeze() */ + MultiXactId new_relminmxid; +} PruneFreezeResult; + +/* 'reason' codes for heap_page_prune_and_freeze() */ typedef enum { PRUNE_ON_ACCESS, /* on-access pruning */ @@ -254,7 +258,7 @@ typedef enum /* * Pruning calculates tuple visibility once and saves the results in an array - * of int8. See PruneResult.htsv for details. This helper function is meant to + * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to * guard against examining visibility status array members which have not yet * been computed. */ @@ -332,6 +336,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, Buffer *buffer, struct TM_FailureData *tmfd); extern void heap_inplace_update(Relation relation, HeapTuple tuple); + extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple, HeapPageFreeze *pagefrz, HeapTupleFreeze *frz, bool *totally_frozen); @@ -358,13 +363,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel, /* in heap/pruneheap.c */ struct GlobalVisState; extern void heap_page_prune_opt(Relation relation, Buffer buffer); -extern void heap_page_prune(Relation relation, Buffer buffer, - struct GlobalVisState *vistest, - bool mark_unused_now, - HeapPageFreeze *pagefrz, - PruneResult *presult, - PruneReason reason, - OffsetNumber *off_loc); +extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer, + struct GlobalVisState *vistest, + bool mark_unused_now, + HeapPageFreeze *pagefrz, + PruneFreezeResult *presult, + PruneReason reason, + OffsetNumber *off_loc); extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only, OffsetNumber *redirected, int nredirected, OffsetNumber *nowdead, int ndead, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 4679660837c..cc6a33ab3ee 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2192,8 +2192,8 @@ ProjectionPath PromptInterruptContext ProtocolVersion PrsStorage +PruneFreezeResult PruneReason -PruneResult PruneState PruneStepResult PsqlScanCallbacks -- 2.40.1 --ck6erxojvlx23byk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch" ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Q: Escapes in jsonpath Idents @ 2024-04-24 19:22 Erik Wienhold <[email protected]> 2024-04-24 20:30 ` Re: Q: Escapes in jsonpath Idents David E. Wheeler <[email protected]> 0 siblings, 1 reply; 24+ messages in thread From: Erik Wienhold @ 2024-04-24 19:22 UTC (permalink / raw) To: David E. Wheeler <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers On 2024-04-24 13:52 +0200, David E. Wheeler wrote: > On Apr 24, 2024, at 05:51, Peter Eisentraut <[email protected]> wrote: > > > A <JSON path identifier> is classified as follows. > > > > Case: > > > > a) A <JSON path identifier> that is a <dollar sign> is a <JSON > > path context variable>. > > > > b) A <JSON path identifier> that begins with <dollar sign> is a > > <JSON path named variable>. > > > > c) Otherwise, a <JSON path identifier> is a <JSON path key name>. > > > > Does this help? I wasn't following all the discussion to see if > > there is anything wrong with the implementation. Thanks Peter! But what is the definition of the entire path expression? Perhaps something like: <JSON path> ::= <JSON path identifier> { "." <JSON path identifier> } That would imply that "$.$foo" is a valid path that accesses a variable member (but I guess the path evaluation is also specified somewhere). Does it say anything about double-quoted accessors? In table 8.25[1] we allow member accessor ."$varname" and it says "If the key name matches some named variable starting with $ or does not meet the JavaScript rules for an identifier, it must be enclosed in double quotes to make it a string literal." What bugs me about this description, after reading it a couple of times, is that it's not clear what is meant by ."$varname". It could mean two things: (1) the double-quoting masks $varname in order to not interpret those characters as a variable or (2) an interpolated string that resolves $varname and yields a dynamic member accessor. The current implementation supports (1), i.e., ."$foo" does not refer to variable foo but the actual property "$foo": => select jsonb_path_query('{"$foo":123,"bar":456}', '$."$foo"', '{"foo":"bar"}'); jsonb_path_query ------------------ 123 (1 row) Under case (2) I'd expect that query to return 456 (because $foo resolves to "bar"). (Similar to how psql would resolve :'foo' to 'bar'.) Variables already work in array accessors and table 8.25 says that "The specified index can be an integer, as well as an expression returning a single numeric value [...]". A variable is such an expression. => select jsonb_path_query('[2,3,5]', '$[$i]', '{"i":1}'); jsonb_path_query ------------------ 3 (1 row) So I'd expect a similar behavior for member accessors as well when seeing ."$varname" in the same table. > Yes, it does, as it ties the special meaning of the dollar sign to the > *beginning* of an expression. So it makes sense that this would be an > error: > > david=# select '$.$foo'::jsonpath; > ERROR: syntax error at or near "$foo" of jsonpath input > LINE 1: select '$.$foo'::jsonpath; > ^ > But I’m less sure when a dollar sign is used in the *middle* (or end) > of a json path identifier: > > david=# select '$.xx$foo'::jsonpath; > ERROR: syntax error at or near "$foo" of jsonpath input > LINE 1: select '$.xx$foo'::jsonpath; > ^ > Perhaps that should be valid? Yes, I think so. That would be case C in the spec excerpt provided by Peter. So it's just a key name that happens to contain (but not start with) the dollar sign. [1] https://www.postgresql.org/docs/current/datatype-json.html#TYPE-JSONPATH-ACCESSORS -- Erik ^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Q: Escapes in jsonpath Idents 2024-04-24 19:22 Re: Q: Escapes in jsonpath Idents Erik Wienhold <[email protected]> @ 2024-04-24 20:30 ` David E. Wheeler <[email protected]> 0 siblings, 0 replies; 24+ messages in thread From: David E. Wheeler @ 2024-04-24 20:30 UTC (permalink / raw) To: Erik Wienhold <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers On Apr 24, 2024, at 3:22 PM, Erik Wienhold <[email protected]> wrote: > Thanks Peter! But what is the definition of the entire path expression? > Perhaps something like: > > <JSON path> ::= <JSON path identifier> { "." <JSON path identifier> } > > That would imply that "$.$foo" is a valid path that accesses a variable > member (but I guess the path evaluation is also specified somewhere). I read it as “if it starts with a dollar sign, it’s a variable and not a path identifier”, and I assume any `.foo` expression is a path identifier. > What bugs me about this description, after reading it a couple of times, > is that it's not clear what is meant by ."$varname". It could mean two > things: (1) the double-quoting masks $varname in order to not interpret > those characters as a variable or (2) an interpolated string that > resolves $varname and yields a dynamic member accessor. My understanding is that if it’s in double quotes it’s never anything other than a string (whether a string literal or a path identifier string literal). IOW, variables don’t interpolate inside strings. > Under case (2) I'd expect that query to return 456 (because $foo > resolves to "bar"). (Similar to how psql would resolve :'foo' to > 'bar'.) Yes, I suspect this is the correct interpretation, but agree the wording could use some massaging, especially since path identifiers cannot start with a dollar sign anyway. Perhaps: "If the key name starts with $ or does not meet the JavaScript rules for an identifier, it must be enclosed in double quotes to make it a string literal." > Variables already work in array accessors and table 8.25 says that "The > specified index can be an integer, as well as an expression returning a > single numeric value [...]". A variable is such an expression. > > => select jsonb_path_query('[2,3,5]', '$[$i]', '{"i":1}'); > jsonb_path_query > ------------------ > 3 > (1 row) > > So I'd expect a similar behavior for member accessors as well when > seeing ."$varname" in the same table. Oh, interesting point! Now I wonder if the standard has this inconsistency (and is aware of it). > Yes, I think so. That would be case C in the spec excerpt provided by > Peter. So it's just a key name that happens to contain (but not start > with) the dollar sign. Exactly. It also matches the doc you quote above. Something would have to change in src/backend/utils/adt/jsonpath_scan.l to fix that, but that file makes my eyes water, so I’m not gonna take a stab at it. :-) D ^ permalink raw reply [nested|flat] 24+ messages in thread
end of thread, other threads:[~2024-04-24 20:30 UTC | newest] Thread overview: 24+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-06-26 09:42 Thinko/typo in ExecSimpleRelationInsert Ashutosh Bapat <[email protected]> 2018-06-26 11:03 ` Ashutosh Bapat <[email protected]> 2018-06-26 12:48 ` Amit Kapila <[email protected]> 2018-06-26 13:32 ` Ashutosh Bapat <[email protected]> 2018-06-26 15:13 ` Amit Kapila <[email protected]> 2018-06-27 04:39 ` Ashutosh Bapat <[email protected]> 2018-06-27 05:54 ` Amit Kapila <[email protected]> 2018-06-27 06:00 ` Ashutosh Bapat <[email protected]> 2018-06-27 13:15 ` Amit Kapila <[email protected]> 2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-26 00:32 [PATCH v7 08/16] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-26 00:32 [PATCH v9 08/21] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-26 00:32 [PATCH v9 08/21] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-03-26 00:32 [PATCH v7 08/16] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]> 2024-04-24 19:22 Re: Q: Escapes in jsonpath Idents Erik Wienhold <[email protected]> 2024-04-24 20:30 ` Re: Q: Escapes in jsonpath Idents David E. Wheeler <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox