agora inbox for [email protected]
help / color / mirror / Atom feedFast default stuff versus pg_upgrade
57+ messages / 9 participants
[nested] [flat]
* Fast default stuff versus pg_upgrade
@ 2018-06-19 14:55 Tom Lane <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-19 14:55 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: [email protected]
AFAICS, the fast-default patch neglected to consider what happens if
a database containing columns with active attmissingval entries is
pg_upgraded. I do not see any code in either pg_dump or pg_upgrade that
attempts to deal with that situation, which means the effect will be
that the "missing" values will silently revert to nulls: they're still
null in the table storage, and the restored pg_attribute entries won't
have anything saying it should be different.
The pg_upgrade regression test fails to exercise such a case. There is
only one table in the ending state of the regression database that has
any atthasmissing columns, and it's empty :-(. If I add a table in
which there actually are active attmissingval entries, say according
to the attached patch, I get a failure in the pg_upgrade test.
This is certainly a stop-ship issue, and in fact it's bad enough
that I think we may need to pull the feature for v11. Designing
binary-upgrade support for this seems like a rather large task
to be starting post-beta1. Nor do I think it's okay to wait for
v12 to make it work; what if we have to force an initdb later in
beta, or recommend use of pg_upgrade for some manual catalog fix
after release?
regards, tom lane
Attachments:
[text/x-diff] ensure-fast-default-gets-tested-in-pg-upgrade.patch (1.4K, ../../[email protected]/2-ensure-fast-default-gets-tested-in-pg-upgrade.patch)
download | inline diff:
diff --git a/src/test/regress/expected/fast_default.out b/src/test/regress/expected/fast_default.out
index ef8d04f..f3d783c 100644
*** a/src/test/regress/expected/fast_default.out
--- b/src/test/regress/expected/fast_default.out
*************** DROP TABLE has_volatile;
*** 548,550 ****
--- 548,561 ----
DROP EVENT TRIGGER has_volatile_rewrite;
DROP FUNCTION log_rewrite;
DROP SCHEMA fast_default;
+ -- Leave a table with an active fast default in place, for pg_upgrade testing
+ set search_path = public;
+ create table has_fast_default(f1 int);
+ insert into has_fast_default values(1);
+ alter table has_fast_default add column f2 int default 42;
+ table has_fast_default;
+ f1 | f2
+ ----+----
+ 1 | 42
+ (1 row)
+
diff --git a/src/test/regress/sql/fast_default.sql b/src/test/regress/sql/fast_default.sql
index 0e66033..7b9cc47 100644
*** a/src/test/regress/sql/fast_default.sql
--- b/src/test/regress/sql/fast_default.sql
*************** DROP TABLE has_volatile;
*** 369,371 ****
--- 369,378 ----
DROP EVENT TRIGGER has_volatile_rewrite;
DROP FUNCTION log_rewrite;
DROP SCHEMA fast_default;
+
+ -- Leave a table with an active fast default in place, for pg_upgrade testing
+ set search_path = public;
+ create table has_fast_default(f1 int);
+ insert into has_fast_default values(1);
+ alter table has_fast_default add column f2 int default 42;
+ table has_fast_default;
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 15:51 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-19 15:51 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: [email protected]
On 06/19/2018 10:55 AM, Tom Lane wrote:
> AFAICS, the fast-default patch neglected to consider what happens if
> a database containing columns with active attmissingval entries is
> pg_upgraded. I do not see any code in either pg_dump or pg_upgrade that
> attempts to deal with that situation, which means the effect will be
> that the "missing" values will silently revert to nulls: they're still
> null in the table storage, and the restored pg_attribute entries won't
> have anything saying it should be different.
>
> The pg_upgrade regression test fails to exercise such a case. There is
> only one table in the ending state of the regression database that has
> any atthasmissing columns, and it's empty :-(. If I add a table in
> which there actually are active attmissingval entries, say according
> to the attached patch, I get a failure in the pg_upgrade test.
>
> This is certainly a stop-ship issue, and in fact it's bad enough
> that I think we may need to pull the feature for v11. Designing
> binary-upgrade support for this seems like a rather large task
> to be starting post-beta1. Nor do I think it's okay to wait for
> v12 to make it work; what if we have to force an initdb later in
> beta, or recommend use of pg_upgrade for some manual catalog fix
> after release?
Ouch!
I guess I have to say mea culpa.
My initial thought was that as a fallback we should disable pg_upgrade
on databases containing such values, and document the limitation in the
docs and the release notes. The workaround would be to force a table
rewrite which would clear them if necessary.
Have we ever recommended use of pg_upgrade for some manual catalog fix
after release? I don't recall doing so. Certainly it hasn't been common.
I have no idea how large an actual fix might be. I'll at least start
working on it immediately. I agree it's very late in the day.
cheers
andrew
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:05 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 2 replies; 57+ messages in thread
From: Andres Freund @ 2018-06-19 16:05 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
Hi,
On 2018-06-19 11:51:16 -0400, Andrew Dunstan wrote:
> My initial thought was that as a fallback we should disable pg_upgrade on
> databases containing such values, and document the limitation in the docs
> and the release notes. The workaround would be to force a table rewrite
> which would clear them if necessary.
I personally would say that that's not acceptable. People will start
using fast defaults - and you can't even do anything against it! - and
suddenly pg_upgrade won't work. But they will only notice that years
later, after collecting terrabytes of data in such tables.
If we can't fix it properly, then imo we should revert / neuter the
feature.
> Have we ever recommended use of pg_upgrade for some manual catalog fix after
> release? I don't recall doing so. Certainly it hasn't been common.
No, but why does it matter? Are you arguing we can delay pg_dump support
for fast defaults to v12?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:08 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-19 16:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
Andres Freund <[email protected]> writes:
> On 2018-06-19 11:51:16 -0400, Andrew Dunstan wrote:
>> Have we ever recommended use of pg_upgrade for some manual catalog fix after
>> release? I don't recall doing so. Certainly it hasn't been common.
> No, but why does it matter?
We absolutely have, as recently as last month:
* Fix incorrect volatility markings on a few built-in functions
(Thomas Munro, Tom Lane)
... can be fixed by manually updating these functions' pg_proc
entries, for example ALTER FUNCTION pg_catalog.query_to_xml(text,
boolean, boolean, text) VOLATILE. (Note that that will need to be
done in each database of the installation.) Another option is to
pg_upgrade the database to a version containing the corrected
initial data.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:17 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-19 16:17 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: [email protected]
Andrew Dunstan <[email protected]> writes:
> I have no idea how large an actual fix might be. I'll at least start
> working on it immediately. I agree it's very late in the day.
On reflection, it seems like there are two moving parts needed:
* Add a binary-upgrade support function to the backend, which would take,
say, table oid, column name, and some representation of the default value;
* Teach pg_dump when operating in binary-upgrade mode to emit a call to
such a function for each column that has atthasmissing true.
The hard part here is how exactly are we going to represent the default
value. AFAICS, the only thing that pg_dump could readily lay its hands
on is the "anyarray" textual representation of attmissingval, which maybe
is okay but it means more work for the support function. Too bad we did
not just store the value in bytea format.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:17 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-19 16:17 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
On 06/19/2018 12:05 PM, Andres Freund wrote:
> Hi,
>
> On 2018-06-19 11:51:16 -0400, Andrew Dunstan wrote:
>> My initial thought was that as a fallback we should disable pg_upgrade on
>> databases containing such values, and document the limitation in the docs
>> and the release notes. The workaround would be to force a table rewrite
>> which would clear them if necessary.
> I personally would say that that's not acceptable. People will start
> using fast defaults - and you can't even do anything against it! - and
> suddenly pg_upgrade won't work. But they will only notice that years
> later, after collecting terrabytes of data in such tables.
Umm, barring the case that Tom mentioned by then it would just work.
It's not the case that if they put in fast default values today they
will never be able to upgrade.
>
> If we can't fix it properly, then imo we should revert / neuter the
> feature.
>
>
>> Have we ever recommended use of pg_upgrade for some manual catalog fix after
>> release? I don't recall doing so. Certainly it hasn't been common.
> No, but why does it matter? Are you arguing we can delay pg_dump support
> for fast defaults to v12?
>
Right now I'm more or less thinking out loud, not arguing anything.
I'd at least like to see what a solution might look like before ruling
it out. I suspect I can come up with something in a day or so. The work
wouldn't be wasted.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:33 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-19 16:33 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
On 2018-06-19 12:17:56 -0400, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
> > I have no idea how large an actual fix might be. I'll at least start
> > working on it immediately. I agree it's very late in the day.
>
> On reflection, it seems like there are two moving parts needed:
>
> * Add a binary-upgrade support function to the backend, which would take,
> say, table oid, column name, and some representation of the default value;
>
> * Teach pg_dump when operating in binary-upgrade mode to emit a call to
> such a function for each column that has atthasmissing true.
>
> The hard part here is how exactly are we going to represent the default
> value. AFAICS, the only thing that pg_dump could readily lay its hands
> on is the "anyarray" textual representation of attmissingval, which maybe
> is okay but it means more work for the support function.
Isn't that just a few lines of code? And if the default value bugs us,
we can easily add a support function that dumps the value without the
anyarray adornment?
> Too bad we did not just store the value in bytea format.
That still seems the right thing to me, not being able in areasonable
way to inspect the default values in the catalog seems worse. We could
have added a new non-array pseudo-type as well, but that's a fair bit of
work...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:36 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Andres Freund @ 2018-06-19 16:36 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; [email protected]
On 2018-06-19 12:17:59 -0400, Andrew Dunstan wrote:
>
>
> On 06/19/2018 12:05 PM, Andres Freund wrote:
> > Hi,
> >
> > On 2018-06-19 11:51:16 -0400, Andrew Dunstan wrote:
> > > My initial thought was that as a fallback we should disable pg_upgrade on
> > > databases containing such values, and document the limitation in the docs
> > > and the release notes. The workaround would be to force a table rewrite
> > > which would clear them if necessary.
> > I personally would say that that's not acceptable. People will start
> > using fast defaults - and you can't even do anything against it! - and
> > suddenly pg_upgrade won't work. But they will only notice that years
> > later, after collecting terrabytes of data in such tables.
>
>
> Umm, barring the case that Tom mentioned by then it would just work.
Huh?
> It's not the case that if they put in fast default values today they
> will never be able to upgrade.
How? I mean upgrading and loosing your default values certainly ain't
ok? And we can't expect users to rewrite their tables, that's why we
added fast default support and why pg_upgrade is used.
> I'd at least like to see what a solution might look like before ruling it
> out. I suspect I can come up with something in a day or so. The work
> wouldn't be wasted.
I think it'd be unacceptable to release v11 without support, but I also
think it's quite possible to just add the necessary logic for v11 if we
put some effort into it. ISTM we've resolved worse issues during beta
than this.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:37 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 3 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-19 16:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
Andres Freund <[email protected]> writes:
> On 2018-06-19 12:17:56 -0400, Tom Lane wrote:
>> The hard part here is how exactly are we going to represent the default
>> value. AFAICS, the only thing that pg_dump could readily lay its hands
>> on is the "anyarray" textual representation of attmissingval, which maybe
>> is okay but it means more work for the support function.
> Isn't that just a few lines of code?
Not sure; I've not thought about how to code it.
> And if the default value bugs us,
> we can easily add a support function that dumps the value without the
> anyarray adornment?
The problem here is that that function does not exist in 11beta1.
Since adding the "incoming" function is certainly going to require
initdb, we have to be able to dump from the server as it now stands,
or we'll be cutting existing beta testers adrift.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 16:52 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
2 siblings, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-19 16:52 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
Hi,
On 2018-06-19 12:37:52 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > And if the default value bugs us,
> > we can easily add a support function that dumps the value without the
> > anyarray adornment?
>
> The problem here is that that function does not exist in 11beta1.
> Since adding the "incoming" function is certainly going to require
> initdb, we have to be able to dump from the server as it now stands,
> or we'll be cutting existing beta testers adrift.
It'd probably not be too hard to write a plpgsql replacement for it,
should it come to that. Obviously it'd be nicer to not require users to
create that, but ...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 17:12 Robert Haas <[email protected]>
parent: Tom Lane <[email protected]>
2 siblings, 1 reply; 57+ messages in thread
From: Robert Haas @ 2018-06-19 17:12 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 19, 2018 at 12:37 PM, Tom Lane <[email protected]> wrote:
> The problem here is that that function does not exist in 11beta1.
> Since adding the "incoming" function is certainly going to require
> initdb, we have to be able to dump from the server as it now stands,
> or we'll be cutting existing beta testers adrift.
That would still be less disruptive than ripping the feature out,
which would be cutting those same users adrift, too, unless I'm
missing something.
I have to admit that I think this feature is scary. I'm not sure that
it was adequately reviewed and tested, and I'm worried this may not be
the only problem it causes. But this particular problem, as Andres
says, doesn't seem like anything we can't fix with acceptable risk.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 17:19 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-19 17:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
Andres Freund <[email protected]> writes:
> On 2018-06-19 12:37:52 -0400, Tom Lane wrote:
>> The problem here is that that function does not exist in 11beta1.
>> Since adding the "incoming" function is certainly going to require
>> initdb, we have to be able to dump from the server as it now stands,
>> or we'll be cutting existing beta testers adrift.
> It'd probably not be too hard to write a plpgsql replacement for it,
> should it come to that. Obviously it'd be nicer to not require users to
> create that, but ...
After some thought, I think it's not that hard to get the support function
to accept the anyarray string form. I was worried about issues like
whether float8 values would restore exactly, but really that's no worse
than a dump/reload today. Basically, the support function would just need
to extract the target attribute's type and typmod from the pg_attribute
row, then call array_in().
I wonder though whether there are any interesting corner cases along
this line:
1. Create a column with a fast default.
2. Sometime later, alter the column so that the fast default value
is no longer a legal value. If the fast default isn't in active use
in the table, the ALTER would go through; but if it does not remove
the attmissingval entry, then ...
3. Subsequently, pg_upgrade fails when the support function tries to
pass the attmissingval entry through the type input function.
The kind of case where this might fail is reducing the allowed
max len (typmod) for a varchar column. I think ALTER TABLE is
smart enough to not rewrite the table for that, so that there
wouldn't be anything causing the fast default to get removed.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 17:23 Peter Geoghegan <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Peter Geoghegan @ 2018-06-19 17:23 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 19, 2018 at 10:12 AM, Robert Haas <[email protected]> wrote:
> I have to admit that I think this feature is scary. I'm not sure that
> it was adequately reviewed and tested, and I'm worried this may not be
> the only problem it causes. But this particular problem, as Andres
> says, doesn't seem like anything we can't fix with acceptable risk.
I agree with both points.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 17:39 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-19 17:39 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Andres Freund <[email protected]>; +Cc: [email protected]
On 06/19/2018 01:19 PM, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
>> On 2018-06-19 12:37:52 -0400, Tom Lane wrote:
>>> The problem here is that that function does not exist in 11beta1.
>>> Since adding the "incoming" function is certainly going to require
>>> initdb, we have to be able to dump from the server as it now stands,
>>> or we'll be cutting existing beta testers adrift.
>> It'd probably not be too hard to write a plpgsql replacement for it,
>> should it come to that. Obviously it'd be nicer to not require users to
>> create that, but ...
> After some thought, I think it's not that hard to get the support function
> to accept the anyarray string form. I was worried about issues like
> whether float8 values would restore exactly, but really that's no worse
> than a dump/reload today. Basically, the support function would just need
> to extract the target attribute's type and typmod from the pg_attribute
> row, then call array_in().
>
> I wonder though whether there are any interesting corner cases along
> this line:
>
> 1. Create a column with a fast default.
>
> 2. Sometime later, alter the column so that the fast default value
> is no longer a legal value. If the fast default isn't in active use
> in the table, the ALTER would go through; but if it does not remove
> the attmissingval entry, then ...
>
> 3. Subsequently, pg_upgrade fails when the support function tries to
> pass the attmissingval entry through the type input function.
>
> The kind of case where this might fail is reducing the allowed
> max len (typmod) for a varchar column. I think ALTER TABLE is
> smart enough to not rewrite the table for that, so that there
> wouldn't be anything causing the fast default to get removed.
>
>
My experimentation showed this causing a rewrite. I think it only skips
the rewrite if you make the allowed length greater, not smaller.
cheers
andrew
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 17:41 David G. Johnston <[email protected]>
parent: Tom Lane <[email protected]>
2 siblings, 1 reply; 57+ messages in thread
From: David G. Johnston @ 2018-06-19 17:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 19, 2018 at 9:37 AM, Tom Lane <[email protected]> wrote:
> The problem here is that that function does not exist in 11beta1.
> Since adding the "incoming" function is certainly going to require
> initdb, we have to be able to dump from the server as it now stands,
> or we'll be cutting existing beta testers adrift.
>
I was under the impression that we don't promise to support a "v10 -> beta
-> rc -> final" upgrade path; instead, once final is released people would
be expected to upgrade "v10 -> v11". Under that condition requiring users
to do "v10 -> beta2" instead of "beta1 -> beta2", while annoying, is well
within the realm of possibility and expectation.
David J.
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-19 17:59 Tom Lane <[email protected]>
parent: David G. Johnston <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-19 17:59 UTC (permalink / raw)
To: David G. Johnston <[email protected]>; +Cc: Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; PostgreSQL Hackers <[email protected]>
"David G. Johnston" <[email protected]> writes:
> On Tue, Jun 19, 2018 at 9:37 AM, Tom Lane <[email protected]> wrote:
>> The problem here is that that function does not exist in 11beta1.
>> Since adding the "incoming" function is certainly going to require
>> initdb, we have to be able to dump from the server as it now stands,
>> or we'll be cutting existing beta testers adrift.
> I was under the impression that we don't promise to support a "v10 -> beta
> -> rc -> final" upgrade path; instead, once final is released people would
> be expected to upgrade "v10 -> v11".
Well, we don't *promise* beta testers that their beta databases will be
usable into production, but ever since pg_upgrade became available we've
tried to make it possible to pg_upgrade to the next beta or production
release. I do not offhand recall any previous case where we failed to do
so.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-20 02:41 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-20 02:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
On 06/19/2018 01:19 PM, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
>> On 2018-06-19 12:37:52 -0400, Tom Lane wrote:
>>> The problem here is that that function does not exist in 11beta1.
>>> Since adding the "incoming" function is certainly going to require
>>> initdb, we have to be able to dump from the server as it now stands,
>>> or we'll be cutting existing beta testers adrift.
>> It'd probably not be too hard to write a plpgsql replacement for it,
>> should it come to that. Obviously it'd be nicer to not require users to
>> create that, but ...
> After some thought, I think it's not that hard to get the support function
> to accept the anyarray string form. I was worried about issues like
> whether float8 values would restore exactly, but really that's no worse
> than a dump/reload today. Basically, the support function would just need
> to extract the target attribute's type and typmod from the pg_attribute
> row, then call array_in().
>
This unfortunately crashes and burns if we use DirectFunctionCall3 to
call array_in, because it uses fn_extra. There is the
CallerFInfoFunctionCall stuff, but it only has 1 and 2 arg variants, and
array_in takes 3. In retrospect we should probably have added a 3 arg
form - quite a few input functions take 3 args. Anything else is likely
to be rather uglier.
Attaching the failing patch. I'll attack this again in the morning.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-patch] fix-default-1.patch (10.6K, ../../[email protected]/2-fix-default-1.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 0c54b02..d9474db 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,14 +11,18 @@
#include "postgres.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
#include "catalog/binary_upgrade.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/extension.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
-
+#include "utils/rel.h"
+#include "utils/syscache.h"
#define CHECK_IS_BINARY_UPGRADE \
do { \
@@ -192,3 +196,59 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+Datum
+binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
+{
+ Oid table_id = PG_GETARG_OID(0);
+ text *attname = PG_GETARG_TEXT_P(1);
+ text *value = PG_GETARG_TEXT_P(2);
+ Datum valuesAtt[Natts_pg_attribute];
+ bool nullsAtt[Natts_pg_attribute];
+ bool replacesAtt[Natts_pg_attribute];
+ Datum missingval;
+ Form_pg_attribute attStruct;
+ Relation attrrel;
+ HeapTuple atttup;
+
+ CHECK_IS_BINARY_UPGRADE;
+
+ /* Lock the attribute row and get the data */
+ attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
+ atttup = SearchSysCacheAttName(table_id,text_to_cstring(attname));
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %s of relation %u",
+ text_to_cstring(attname), table_id);
+ attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
+
+
+ /* get an array value from the value string */
+ missingval = DirectFunctionCall3(array_in,
+ CStringGetDatum(text_to_cstring(value)),
+ ObjectIdGetDatum(attStruct->atttypid),
+ Int32GetDatum(attStruct->atttypmod));
+
+ /* update the tuple - set atthasmissing and attmissingval */
+ MemSet(valuesAtt, 0, sizeof(valuesAtt));
+ MemSet(nullsAtt, false, sizeof(nullsAtt));
+ MemSet(replacesAtt, false, sizeof(replacesAtt));
+
+ valuesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
+ replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
+ nullsAtt[Anum_pg_attribute_attmissingval - 1] = false;
+
+ atttup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
+ valuesAtt, nullsAtt, replacesAtt);
+ CatalogTupleUpdate(attrrel, &atttup->t_self, atttup);
+
+
+ /* clean up */
+ pfree(DatumGetPointer(missingval));
+
+ heap_close(attrrel, RowExclusiveLock);
+ heap_freetuple(atttup);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea2f022..22be3c2 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8095,6 +8095,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_typstorage;
int i_attnotnull;
int i_atthasdef;
+ int i_atthasmissing;
int i_attidentity;
int i_attisdropped;
int i_attlen;
@@ -8103,6 +8104,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_attoptions;
int i_attcollation;
int i_attfdwoptions;
+ int i_attmissingval;
PGresult *res;
int ntups;
bool hasdefaults;
@@ -8132,7 +8134,33 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
resetPQExpBuffer(q);
- if (fout->remoteVersion >= 100000)
+ if (fout->remoteVersion >= 110000)
+ {
+ /* atthasmissing and attmissingval are new in 11 */
+ appendPQExpBuffer(q, "SELECT a.attnum, a.attname, a.atttypmod, "
+ "a.attstattarget, a.attstorage, t.typstorage, "
+ "a.attnotnull, a.atthasdef, a.attisdropped, "
+ "a.attlen, a.attalign, a.attislocal, "
+ "pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
+ "array_to_string(a.attoptions, ', ') AS attoptions, "
+ "CASE WHEN a.attcollation <> t.typcollation "
+ "THEN a.attcollation ELSE 0 END AS attcollation, "
+ "a.atthasmissing, a.attidentity, "
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(attfdwoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attfdwoptions ,"
+ "a.attmissingval "
+ "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
+ "ON a.atttypid = t.oid "
+ "WHERE a.attrelid = '%u'::pg_catalog.oid "
+ "AND a.attnum > 0::pg_catalog.int2 "
+ "ORDER BY a.attnum",
+ tbinfo->dobj.catId.oid);
+ }
+ else if (fout->remoteVersion >= 100000)
{
/*
* attidentity is new in version 10.
@@ -8258,6 +8286,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_typstorage = PQfnumber(res, "typstorage");
i_attnotnull = PQfnumber(res, "attnotnull");
i_atthasdef = PQfnumber(res, "atthasdef");
+ i_atthasmissing = PQfnumber(res, "atthasmissing");
i_attidentity = PQfnumber(res, "attidentity");
i_attisdropped = PQfnumber(res, "attisdropped");
i_attlen = PQfnumber(res, "attlen");
@@ -8266,6 +8295,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_attoptions = PQfnumber(res, "attoptions");
i_attcollation = PQfnumber(res, "attcollation");
i_attfdwoptions = PQfnumber(res, "attfdwoptions");
+ i_attmissingval = PQfnumber(res, "attmissingval");
tbinfo->numatts = ntups;
tbinfo->attnames = (char **) pg_malloc(ntups * sizeof(char *));
@@ -8274,6 +8304,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget = (int *) pg_malloc(ntups * sizeof(int));
tbinfo->attstorage = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->typstorage = (char *) pg_malloc(ntups * sizeof(char));
+ tbinfo->atthasmissing = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attidentity = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->attisdropped = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attlen = (int *) pg_malloc(ntups * sizeof(int));
@@ -8282,6 +8313,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8299,6 +8331,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget[j] = atoi(PQgetvalue(res, j, i_attstattarget));
tbinfo->attstorage[j] = *(PQgetvalue(res, j, i_attstorage));
tbinfo->typstorage[j] = *(PQgetvalue(res, j, i_typstorage));
+ tbinfo->atthasmissing[j] = (i_atthasmissing >= 0 ? (PQgetvalue(res, j, i_atthasmissing)[0] == 't') : false);
tbinfo->attidentity[j] = (i_attidentity >= 0 ? *(PQgetvalue(res, j, i_attidentity)) : '\0');
tbinfo->needs_override = tbinfo->needs_override || (tbinfo->attidentity[j] == ATTRIBUTE_IDENTITY_ALWAYS);
tbinfo->attisdropped[j] = (PQgetvalue(res, j, i_attisdropped)[0] == 't');
@@ -8309,6 +8342,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, j, i_attoptions));
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, i_attcollation));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, i_attfdwoptions));
+ tbinfo->attmissingval[j] = (tbinfo->atthasmissing[j] ? pg_strdup(PQgetvalue(res, j, i_attmissingval)) : NULL );
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, i_atthasdef)[0] == 't')
hasdefaults = true;
@@ -15659,6 +15693,29 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
appendPQExpBufferStr(q, ";\n");
/*
+ * in binary upgrade mode, update the catalog with any missing values
+ * that might be present.
+ */
+ if (dopt->binary_upgrade)
+ {
+ for (j = 0; j < tbinfo->numatts; j++)
+ {
+ if (tbinfo->atthasmissing[j])
+ {
+ appendPQExpBufferStr(q, "\n-- set missing value.\n");
+ appendPQExpBufferStr(q,
+ "SELECT pg_catalog.binary_upgrade_set_missing_value(");
+ appendStringLiteralAH(q,qualrelname, fout);
+ appendPQExpBufferStr(q, "::pg_catalog.regclass,");
+ appendStringLiteralAH(q, tbinfo->attnames[j], fout);
+ appendPQExpBufferStr(q,",");
+ appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
+ appendPQExpBufferStr(q,");\n\n");
+ }
+ }
+ }
+
+ /*
* To create binary-compatible heap files, we have to ensure the same
* physical column order, including dropped columns, as in the
* original. Therefore, we create dropped columns above and drop them
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e96c662..c1681b3 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -309,6 +309,7 @@ typedef struct _tableInfo
char *attstorage; /* attribute storage scheme */
char *typstorage; /* type storage scheme */
bool *attisdropped; /* true if attr is dropped; don't dump it */
+ bool *atthasmissing; /* true if the attribute has a missing value */
char *attidentity;
int *attlen; /* attribute length, used by binary_upgrade */
char *attalign; /* attribute align, used by binary_upgrade */
@@ -316,6 +317,7 @@ typedef struct _tableInfo
char **attoptions; /* per-attribute options */
Oid *attcollation; /* per-attribute collation selection */
char **attfdwoptions; /* per-attribute fdw options */
+ char **attmissingval; /* per attribute missing value */
bool *notnull; /* NOT NULL constraints on attributes */
bool *inhNotNull; /* true if NOT NULL is inherited */
struct _attrDefInfo **attrdefs; /* DEFAULT expressions */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66c6c22..9834e38 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10037,6 +10037,10 @@
proname => 'binary_upgrade_set_record_init_privs', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'bool',
prosrc => 'binary_upgrade_set_record_init_privs' },
+{ oid => '4101', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_missing_value', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid text text',
+ prosrc => 'binary_upgrade_set_missing_value' },
# replication/origin.h
{ oid => '6003', descr => 'create a replication origin',
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-20 02:46 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-20 02:46 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; [email protected]
On 2018-06-19 22:41:26 -0400, Andrew Dunstan wrote:
> This unfortunately crashes and burns if we use DirectFunctionCall3 to call
> array_in, because it uses fn_extra. There is the CallerFInfoFunctionCall
> stuff, but it only has 1 and 2 arg variants, and array_in takes 3. In
> retrospect we should probably have added a 3 arg form - quite a few input
> functions take 3 args. Anything else is likely to be rather uglier.
>
> Attaching the failing patch. I'll attack this again in the morning.
Why don't you just use OidFunctionCall3? Or simply an explicit
fmgr_info(), InitFunctionCallInfoData(), FunctionCallInvoke() combo?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-20 16:51 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-20 16:51 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
On 06/19/2018 10:46 PM, Andres Freund wrote:
> On 2018-06-19 22:41:26 -0400, Andrew Dunstan wrote:
>> This unfortunately crashes and burns if we use DirectFunctionCall3 to call
>> array_in, because it uses fn_extra. There is the CallerFInfoFunctionCall
>> stuff, but it only has 1 and 2 arg variants, and array_in takes 3. In
>> retrospect we should probably have added a 3 arg form - quite a few input
>> functions take 3 args. Anything else is likely to be rather uglier.
>>
>> Attaching the failing patch. I'll attack this again in the morning.
> Why don't you just use OidFunctionCall3? Or simply an explicit
> fmgr_info(), InitFunctionCallInfoData(), FunctionCallInvoke() combo?
>
Thanks for that. I should not code late at night.
Here's a version that works in my testing with Tom's patch making sure
there's a missing value to migrate applied. Thanks to Alvaro for some
useful criticism - any errors are of course my responsibility.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-patch] fix-default-3.patch (10.9K, ../../[email protected]/2-fix-default-3.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 0c54b02..2666ab2 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,14 +11,19 @@
#include "postgres.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
#include "catalog/binary_upgrade.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/extension.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
-
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
#define CHECK_IS_BINARY_UPGRADE \
do { \
@@ -192,3 +197,66 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+Datum
+binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
+{
+ Oid table_id = PG_GETARG_OID(0);
+ text *attname = PG_GETARG_TEXT_P(1);
+ text *value = PG_GETARG_TEXT_P(2);
+ Datum valuesAtt[Natts_pg_attribute];
+ bool nullsAtt[Natts_pg_attribute];
+ bool replacesAtt[Natts_pg_attribute];
+ Datum missingval;
+ Form_pg_attribute attStruct;
+ Relation attrrel;
+ HeapTuple atttup, newtup;
+ Oid inputfunc, inputparam;
+ char *cattname = text_to_cstring(attname);
+ char *cvalue = text_to_cstring(value);
+
+ CHECK_IS_BINARY_UPGRADE;
+
+ /* Lock the attribute row and get the data */
+ attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
+ atttup = SearchSysCacheAttName(table_id,cattname);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %s of relation %u",
+ cattname, table_id);
+ attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /* get an array value from the value string */
+
+ /* find the io info for an arbitrary array type to get array_in Oid */
+ getTypeInputInfo(INT4ARRAYOID, &inputfunc, &inputparam);
+ missingval = OidFunctionCall3(
+ inputfunc, /* array_in */
+ CStringGetDatum(cvalue),
+ ObjectIdGetDatum(attStruct->atttypid),
+ Int32GetDatum(attStruct->atttypmod));
+
+ /* update the tuple - set atthasmissing and attmissingval */
+ MemSet(valuesAtt, 0, sizeof(valuesAtt));
+ MemSet(nullsAtt, false, sizeof(nullsAtt));
+ MemSet(replacesAtt, false, sizeof(replacesAtt));
+
+ valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
+ replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
+ replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
+
+ newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
+ valuesAtt, nullsAtt, replacesAtt);
+ CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
+
+ /* clean up */
+ heap_freetuple(newtup);
+ ReleaseSysCache(atttup);
+ pfree(cattname);
+ pfree(cvalue);
+ pfree(DatumGetPointer(missingval));
+
+ heap_close(attrrel, RowExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea2f022..22be3c2 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8095,6 +8095,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_typstorage;
int i_attnotnull;
int i_atthasdef;
+ int i_atthasmissing;
int i_attidentity;
int i_attisdropped;
int i_attlen;
@@ -8103,6 +8104,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_attoptions;
int i_attcollation;
int i_attfdwoptions;
+ int i_attmissingval;
PGresult *res;
int ntups;
bool hasdefaults;
@@ -8132,7 +8134,33 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
resetPQExpBuffer(q);
- if (fout->remoteVersion >= 100000)
+ if (fout->remoteVersion >= 110000)
+ {
+ /* atthasmissing and attmissingval are new in 11 */
+ appendPQExpBuffer(q, "SELECT a.attnum, a.attname, a.atttypmod, "
+ "a.attstattarget, a.attstorage, t.typstorage, "
+ "a.attnotnull, a.atthasdef, a.attisdropped, "
+ "a.attlen, a.attalign, a.attislocal, "
+ "pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
+ "array_to_string(a.attoptions, ', ') AS attoptions, "
+ "CASE WHEN a.attcollation <> t.typcollation "
+ "THEN a.attcollation ELSE 0 END AS attcollation, "
+ "a.atthasmissing, a.attidentity, "
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(attfdwoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attfdwoptions ,"
+ "a.attmissingval "
+ "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
+ "ON a.atttypid = t.oid "
+ "WHERE a.attrelid = '%u'::pg_catalog.oid "
+ "AND a.attnum > 0::pg_catalog.int2 "
+ "ORDER BY a.attnum",
+ tbinfo->dobj.catId.oid);
+ }
+ else if (fout->remoteVersion >= 100000)
{
/*
* attidentity is new in version 10.
@@ -8258,6 +8286,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_typstorage = PQfnumber(res, "typstorage");
i_attnotnull = PQfnumber(res, "attnotnull");
i_atthasdef = PQfnumber(res, "atthasdef");
+ i_atthasmissing = PQfnumber(res, "atthasmissing");
i_attidentity = PQfnumber(res, "attidentity");
i_attisdropped = PQfnumber(res, "attisdropped");
i_attlen = PQfnumber(res, "attlen");
@@ -8266,6 +8295,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_attoptions = PQfnumber(res, "attoptions");
i_attcollation = PQfnumber(res, "attcollation");
i_attfdwoptions = PQfnumber(res, "attfdwoptions");
+ i_attmissingval = PQfnumber(res, "attmissingval");
tbinfo->numatts = ntups;
tbinfo->attnames = (char **) pg_malloc(ntups * sizeof(char *));
@@ -8274,6 +8304,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget = (int *) pg_malloc(ntups * sizeof(int));
tbinfo->attstorage = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->typstorage = (char *) pg_malloc(ntups * sizeof(char));
+ tbinfo->atthasmissing = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attidentity = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->attisdropped = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attlen = (int *) pg_malloc(ntups * sizeof(int));
@@ -8282,6 +8313,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8299,6 +8331,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget[j] = atoi(PQgetvalue(res, j, i_attstattarget));
tbinfo->attstorage[j] = *(PQgetvalue(res, j, i_attstorage));
tbinfo->typstorage[j] = *(PQgetvalue(res, j, i_typstorage));
+ tbinfo->atthasmissing[j] = (i_atthasmissing >= 0 ? (PQgetvalue(res, j, i_atthasmissing)[0] == 't') : false);
tbinfo->attidentity[j] = (i_attidentity >= 0 ? *(PQgetvalue(res, j, i_attidentity)) : '\0');
tbinfo->needs_override = tbinfo->needs_override || (tbinfo->attidentity[j] == ATTRIBUTE_IDENTITY_ALWAYS);
tbinfo->attisdropped[j] = (PQgetvalue(res, j, i_attisdropped)[0] == 't');
@@ -8309,6 +8342,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, j, i_attoptions));
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, i_attcollation));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, i_attfdwoptions));
+ tbinfo->attmissingval[j] = (tbinfo->atthasmissing[j] ? pg_strdup(PQgetvalue(res, j, i_attmissingval)) : NULL );
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, i_atthasdef)[0] == 't')
hasdefaults = true;
@@ -15659,6 +15693,29 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
appendPQExpBufferStr(q, ";\n");
/*
+ * in binary upgrade mode, update the catalog with any missing values
+ * that might be present.
+ */
+ if (dopt->binary_upgrade)
+ {
+ for (j = 0; j < tbinfo->numatts; j++)
+ {
+ if (tbinfo->atthasmissing[j])
+ {
+ appendPQExpBufferStr(q, "\n-- set missing value.\n");
+ appendPQExpBufferStr(q,
+ "SELECT pg_catalog.binary_upgrade_set_missing_value(");
+ appendStringLiteralAH(q,qualrelname, fout);
+ appendPQExpBufferStr(q, "::pg_catalog.regclass,");
+ appendStringLiteralAH(q, tbinfo->attnames[j], fout);
+ appendPQExpBufferStr(q,",");
+ appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
+ appendPQExpBufferStr(q,");\n\n");
+ }
+ }
+ }
+
+ /*
* To create binary-compatible heap files, we have to ensure the same
* physical column order, including dropped columns, as in the
* original. Therefore, we create dropped columns above and drop them
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e96c662..c1681b3 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -309,6 +309,7 @@ typedef struct _tableInfo
char *attstorage; /* attribute storage scheme */
char *typstorage; /* type storage scheme */
bool *attisdropped; /* true if attr is dropped; don't dump it */
+ bool *atthasmissing; /* true if the attribute has a missing value */
char *attidentity;
int *attlen; /* attribute length, used by binary_upgrade */
char *attalign; /* attribute align, used by binary_upgrade */
@@ -316,6 +317,7 @@ typedef struct _tableInfo
char **attoptions; /* per-attribute options */
Oid *attcollation; /* per-attribute collation selection */
char **attfdwoptions; /* per-attribute fdw options */
+ char **attmissingval; /* per attribute missing value */
bool *notnull; /* NOT NULL constraints on attributes */
bool *inhNotNull; /* true if NOT NULL is inherited */
struct _attrDefInfo **attrdefs; /* DEFAULT expressions */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66c6c22..9834e38 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10037,6 +10037,10 @@
proname => 'binary_upgrade_set_record_init_privs', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'bool',
prosrc => 'binary_upgrade_set_record_init_privs' },
+{ oid => '4101', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_missing_value', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid text text',
+ prosrc => 'binary_upgrade_set_missing_value' },
# replication/origin.h
{ oid => '6003', descr => 'create a replication origin',
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-20 16:58 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-20 16:58 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
Hi,
Just a quick scan-through review:
On 2018-06-20 12:51:07 -0400, Andrew Dunstan wrote:
> diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
> index 0c54b02..2666ab2 100644
> --- a/src/backend/utils/adt/pg_upgrade_support.c
> +++ b/src/backend/utils/adt/pg_upgrade_support.c
> @@ -11,14 +11,19 @@
>
> #include "postgres.h"
>
> +#include "access/heapam.h"
> +#include "access/htup_details.h"
> #include "catalog/binary_upgrade.h"
> +#include "catalog/indexing.h"
> #include "catalog/namespace.h"
> #include "catalog/pg_type.h"
> #include "commands/extension.h"
> #include "miscadmin.h"
> #include "utils/array.h"
> #include "utils/builtins.h"
> -
> +#include "utils/lsyscache.h"
> +#include "utils/rel.h"
> +#include "utils/syscache.h"
>
Seems to delete a newline.
> #define CHECK_IS_BINARY_UPGRADE \
> do { \
> @@ -192,3 +197,66 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
>
> PG_RETURN_VOID();
> }
> +
> +Datum
> +binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
> +{
> + Oid table_id = PG_GETARG_OID(0);
> + text *attname = PG_GETARG_TEXT_P(1);
> + text *value = PG_GETARG_TEXT_P(2);
> + Datum valuesAtt[Natts_pg_attribute];
> + bool nullsAtt[Natts_pg_attribute];
> + bool replacesAtt[Natts_pg_attribute];
> + Datum missingval;
> + Form_pg_attribute attStruct;
> + Relation attrrel;
> + HeapTuple atttup, newtup;
> + Oid inputfunc, inputparam;
> + char *cattname = text_to_cstring(attname);
> + char *cvalue = text_to_cstring(value);
> +
> + CHECK_IS_BINARY_UPGRADE;
> +
> + /* Lock the attribute row and get the data */
> + attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
Maybe I'm being paranoid, but I feel like the relation should also be
locked here.
> + atttup = SearchSysCacheAttName(table_id,cattname);
Missing space before 'cattname'.
> + if (!HeapTupleIsValid(atttup))
> + elog(ERROR, "cache lookup failed for attribute %s of relation %u",
> + cattname, table_id);
> + attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
> +
> + /* get an array value from the value string */
> +
> + /* find the io info for an arbitrary array type to get array_in Oid */
> + getTypeInputInfo(INT4ARRAYOID, &inputfunc, &inputparam);
Maybe I'm confused, but why INT4ARRAYOID? If you're just looking for the
oid of array_in, why not use F_ARRAY_IN?
> + missingval = OidFunctionCall3(
> + inputfunc, /* array_in */
> + CStringGetDatum(cvalue),
> + ObjectIdGetDatum(attStruct->atttypid),
> + Int32GetDatum(attStruct->atttypmod));
> +
> + /* update the tuple - set atthasmissing and attmissingval */
> + MemSet(valuesAtt, 0, sizeof(valuesAtt));
> + MemSet(nullsAtt, false, sizeof(nullsAtt));
> + MemSet(replacesAtt, false, sizeof(replacesAtt));
> +
> + valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
> + replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
> + valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
> + replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
> +
> + newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
> + valuesAtt, nullsAtt, replacesAtt);
> + CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
> + /* clean up */
> + heap_freetuple(newtup);
> + ReleaseSysCache(atttup);
> + pfree(cattname);
> + pfree(cvalue);
> + pfree(DatumGetPointer(missingval));
Is this worth bothering with (except the ReleaseSysCache)? We'll clean
up via memory context soon, no?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-20 17:46 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-20 17:46 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
On 06/20/2018 12:58 PM, Andres Freund wrote:
> Hi,
>
> Just a quick scan-through review:
>
> On 2018-06-20 12:51:07 -0400, Andrew Dunstan wrote:
>> diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
>> index 0c54b02..2666ab2 100644
>> --- a/src/backend/utils/adt/pg_upgrade_support.c
>> +++ b/src/backend/utils/adt/pg_upgrade_support.c
>> @@ -11,14 +11,19 @@
>>
>> #include "postgres.h"
>>
>> +#include "access/heapam.h"
>> +#include "access/htup_details.h"
>> #include "catalog/binary_upgrade.h"
>> +#include "catalog/indexing.h"
>> #include "catalog/namespace.h"
>> #include "catalog/pg_type.h"
>> #include "commands/extension.h"
>> #include "miscadmin.h"
>> #include "utils/array.h"
>> #include "utils/builtins.h"
>> -
>> +#include "utils/lsyscache.h"
>> +#include "utils/rel.h"
>> +#include "utils/syscache.h"
>>
> Seems to delete a newline.
Will fix
>
>
>> #define CHECK_IS_BINARY_UPGRADE \
>> do { \
>> @@ -192,3 +197,66 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
>>
>> PG_RETURN_VOID();
>> }
>> +
>> +Datum
>> +binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
>> +{
>> + Oid table_id = PG_GETARG_OID(0);
>> + text *attname = PG_GETARG_TEXT_P(1);
>> + text *value = PG_GETARG_TEXT_P(2);
>> + Datum valuesAtt[Natts_pg_attribute];
>> + bool nullsAtt[Natts_pg_attribute];
>> + bool replacesAtt[Natts_pg_attribute];
>> + Datum missingval;
>> + Form_pg_attribute attStruct;
>> + Relation attrrel;
>> + HeapTuple atttup, newtup;
>> + Oid inputfunc, inputparam;
>> + char *cattname = text_to_cstring(attname);
>> + char *cvalue = text_to_cstring(value);
>> +
>> + CHECK_IS_BINARY_UPGRADE;
>> +
>> + /* Lock the attribute row and get the data */
>> + attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
> Maybe I'm being paranoid, but I feel like the relation should also be
> locked here.
I wondered about that. Other opinions?
>
>
>> + atttup = SearchSysCacheAttName(table_id,cattname);
> Missing space before 'cattname'.
Will fix.
>
>> + if (!HeapTupleIsValid(atttup))
>> + elog(ERROR, "cache lookup failed for attribute %s of relation %u",
>> + cattname, table_id);
>> + attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
>> +
>> + /* get an array value from the value string */
>> +
>> + /* find the io info for an arbitrary array type to get array_in Oid */
>> + getTypeInputInfo(INT4ARRAYOID, &inputfunc, &inputparam);
> Maybe I'm confused, but why INT4ARRAYOID? If you're just looking for the
> oid of array_in, why not use F_ARRAY_IN?
Brain fade? Will fix.
>
>> + missingval = OidFunctionCall3(
>> + inputfunc, /* array_in */
>> + CStringGetDatum(cvalue),
>> + ObjectIdGetDatum(attStruct->atttypid),
>> + Int32GetDatum(attStruct->atttypmod));
>> +
>> + /* update the tuple - set atthasmissing and attmissingval */
>> + MemSet(valuesAtt, 0, sizeof(valuesAtt));
>> + MemSet(nullsAtt, false, sizeof(nullsAtt));
>> + MemSet(replacesAtt, false, sizeof(replacesAtt));
>> +
>> + valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
>> + replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
>> + valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
>> + replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
>> +
>> + newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
>> + valuesAtt, nullsAtt, replacesAtt);
>> + CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
>> + /* clean up */
>> + heap_freetuple(newtup);
>> + ReleaseSysCache(atttup);
>> + pfree(cattname);
>> + pfree(cvalue);
>> + pfree(DatumGetPointer(missingval));
> Is this worth bothering with (except the ReleaseSysCache)? We'll clean
> up via memory context soon, no?
>
Done from an abundance of caution. I'll remove them.
Thanks for the quick review.
Attached deals with all those issues except locking.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-patch] fix-default-4.patch (10.7K, ../../[email protected]/2-fix-default-4.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 0c54b02..aed85a3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,13 +11,20 @@
#include "postgres.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
#include "catalog/binary_upgrade.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/extension.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -192,3 +199,59 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+Datum
+binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
+{
+ Oid table_id = PG_GETARG_OID(0);
+ text *attname = PG_GETARG_TEXT_P(1);
+ text *value = PG_GETARG_TEXT_P(2);
+ Datum valuesAtt[Natts_pg_attribute];
+ bool nullsAtt[Natts_pg_attribute];
+ bool replacesAtt[Natts_pg_attribute];
+ Datum missingval;
+ Form_pg_attribute attStruct;
+ Relation attrrel;
+ HeapTuple atttup, newtup;
+ char *cattname = text_to_cstring(attname);
+ char *cvalue = text_to_cstring(value);
+
+ CHECK_IS_BINARY_UPGRADE;
+
+ /* Lock the attribute row and get the data */
+ attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
+ atttup = SearchSysCacheAttName(table_id, cattname);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %s of relation %u",
+ cattname, table_id);
+ attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /* get an array value from the value string */
+
+ /* find the io info for an arbitrary array type to get array_in Oid */
+ missingval = OidFunctionCall3(
+ F_ARRAY_IN,
+ CStringGetDatum(cvalue),
+ ObjectIdGetDatum(attStruct->atttypid),
+ Int32GetDatum(attStruct->atttypmod));
+
+ /* update the tuple - set atthasmissing and attmissingval */
+ MemSet(valuesAtt, 0, sizeof(valuesAtt));
+ MemSet(nullsAtt, false, sizeof(nullsAtt));
+ MemSet(replacesAtt, false, sizeof(replacesAtt));
+
+ valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
+ replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
+ replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
+
+ newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
+ valuesAtt, nullsAtt, replacesAtt);
+ CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
+
+ /* clean up */
+ ReleaseSysCache(atttup);
+ heap_close(attrrel, RowExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea2f022..22be3c2 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8095,6 +8095,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_typstorage;
int i_attnotnull;
int i_atthasdef;
+ int i_atthasmissing;
int i_attidentity;
int i_attisdropped;
int i_attlen;
@@ -8103,6 +8104,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_attoptions;
int i_attcollation;
int i_attfdwoptions;
+ int i_attmissingval;
PGresult *res;
int ntups;
bool hasdefaults;
@@ -8132,7 +8134,33 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
resetPQExpBuffer(q);
- if (fout->remoteVersion >= 100000)
+ if (fout->remoteVersion >= 110000)
+ {
+ /* atthasmissing and attmissingval are new in 11 */
+ appendPQExpBuffer(q, "SELECT a.attnum, a.attname, a.atttypmod, "
+ "a.attstattarget, a.attstorage, t.typstorage, "
+ "a.attnotnull, a.atthasdef, a.attisdropped, "
+ "a.attlen, a.attalign, a.attislocal, "
+ "pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
+ "array_to_string(a.attoptions, ', ') AS attoptions, "
+ "CASE WHEN a.attcollation <> t.typcollation "
+ "THEN a.attcollation ELSE 0 END AS attcollation, "
+ "a.atthasmissing, a.attidentity, "
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(attfdwoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attfdwoptions ,"
+ "a.attmissingval "
+ "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
+ "ON a.atttypid = t.oid "
+ "WHERE a.attrelid = '%u'::pg_catalog.oid "
+ "AND a.attnum > 0::pg_catalog.int2 "
+ "ORDER BY a.attnum",
+ tbinfo->dobj.catId.oid);
+ }
+ else if (fout->remoteVersion >= 100000)
{
/*
* attidentity is new in version 10.
@@ -8258,6 +8286,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_typstorage = PQfnumber(res, "typstorage");
i_attnotnull = PQfnumber(res, "attnotnull");
i_atthasdef = PQfnumber(res, "atthasdef");
+ i_atthasmissing = PQfnumber(res, "atthasmissing");
i_attidentity = PQfnumber(res, "attidentity");
i_attisdropped = PQfnumber(res, "attisdropped");
i_attlen = PQfnumber(res, "attlen");
@@ -8266,6 +8295,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_attoptions = PQfnumber(res, "attoptions");
i_attcollation = PQfnumber(res, "attcollation");
i_attfdwoptions = PQfnumber(res, "attfdwoptions");
+ i_attmissingval = PQfnumber(res, "attmissingval");
tbinfo->numatts = ntups;
tbinfo->attnames = (char **) pg_malloc(ntups * sizeof(char *));
@@ -8274,6 +8304,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget = (int *) pg_malloc(ntups * sizeof(int));
tbinfo->attstorage = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->typstorage = (char *) pg_malloc(ntups * sizeof(char));
+ tbinfo->atthasmissing = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attidentity = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->attisdropped = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attlen = (int *) pg_malloc(ntups * sizeof(int));
@@ -8282,6 +8313,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8299,6 +8331,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget[j] = atoi(PQgetvalue(res, j, i_attstattarget));
tbinfo->attstorage[j] = *(PQgetvalue(res, j, i_attstorage));
tbinfo->typstorage[j] = *(PQgetvalue(res, j, i_typstorage));
+ tbinfo->atthasmissing[j] = (i_atthasmissing >= 0 ? (PQgetvalue(res, j, i_atthasmissing)[0] == 't') : false);
tbinfo->attidentity[j] = (i_attidentity >= 0 ? *(PQgetvalue(res, j, i_attidentity)) : '\0');
tbinfo->needs_override = tbinfo->needs_override || (tbinfo->attidentity[j] == ATTRIBUTE_IDENTITY_ALWAYS);
tbinfo->attisdropped[j] = (PQgetvalue(res, j, i_attisdropped)[0] == 't');
@@ -8309,6 +8342,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, j, i_attoptions));
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, i_attcollation));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, i_attfdwoptions));
+ tbinfo->attmissingval[j] = (tbinfo->atthasmissing[j] ? pg_strdup(PQgetvalue(res, j, i_attmissingval)) : NULL );
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, i_atthasdef)[0] == 't')
hasdefaults = true;
@@ -15659,6 +15693,29 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
appendPQExpBufferStr(q, ";\n");
/*
+ * in binary upgrade mode, update the catalog with any missing values
+ * that might be present.
+ */
+ if (dopt->binary_upgrade)
+ {
+ for (j = 0; j < tbinfo->numatts; j++)
+ {
+ if (tbinfo->atthasmissing[j])
+ {
+ appendPQExpBufferStr(q, "\n-- set missing value.\n");
+ appendPQExpBufferStr(q,
+ "SELECT pg_catalog.binary_upgrade_set_missing_value(");
+ appendStringLiteralAH(q,qualrelname, fout);
+ appendPQExpBufferStr(q, "::pg_catalog.regclass,");
+ appendStringLiteralAH(q, tbinfo->attnames[j], fout);
+ appendPQExpBufferStr(q,",");
+ appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
+ appendPQExpBufferStr(q,");\n\n");
+ }
+ }
+ }
+
+ /*
* To create binary-compatible heap files, we have to ensure the same
* physical column order, including dropped columns, as in the
* original. Therefore, we create dropped columns above and drop them
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e96c662..c1681b3 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -309,6 +309,7 @@ typedef struct _tableInfo
char *attstorage; /* attribute storage scheme */
char *typstorage; /* type storage scheme */
bool *attisdropped; /* true if attr is dropped; don't dump it */
+ bool *atthasmissing; /* true if the attribute has a missing value */
char *attidentity;
int *attlen; /* attribute length, used by binary_upgrade */
char *attalign; /* attribute align, used by binary_upgrade */
@@ -316,6 +317,7 @@ typedef struct _tableInfo
char **attoptions; /* per-attribute options */
Oid *attcollation; /* per-attribute collation selection */
char **attfdwoptions; /* per-attribute fdw options */
+ char **attmissingval; /* per attribute missing value */
bool *notnull; /* NOT NULL constraints on attributes */
bool *inhNotNull; /* true if NOT NULL is inherited */
struct _attrDefInfo **attrdefs; /* DEFAULT expressions */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66c6c22..9834e38 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10037,6 +10037,10 @@
proname => 'binary_upgrade_set_record_init_privs', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'bool',
prosrc => 'binary_upgrade_set_record_init_privs' },
+{ oid => '4101', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_missing_value', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid text text',
+ prosrc => 'binary_upgrade_set_missing_value' },
# replication/origin.h
{ oid => '6003', descr => 'create a replication origin',
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 00:53 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 00:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
On 06/20/2018 01:46 PM, Andrew Dunstan wrote:
>
>
>
> Attached deals with all those issues except locking.
>
>
This version adds a lock on the table owning the attribute.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-patch] fix-default-5.patch (10.8K, ../../[email protected]/2-fix-default-5.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 0c54b02..2480fa0 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,13 +11,20 @@
#include "postgres.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
#include "catalog/binary_upgrade.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/extension.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -192,3 +199,63 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+Datum
+binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
+{
+ Oid table_id = PG_GETARG_OID(0);
+ text *attname = PG_GETARG_TEXT_P(1);
+ text *value = PG_GETARG_TEXT_P(2);
+ Datum valuesAtt[Natts_pg_attribute];
+ bool nullsAtt[Natts_pg_attribute];
+ bool replacesAtt[Natts_pg_attribute];
+ Datum missingval;
+ Form_pg_attribute attStruct;
+ Relation attrrel, tablerel;
+ HeapTuple atttup, newtup;
+ char *cattname = text_to_cstring(attname);
+ char *cvalue = text_to_cstring(value);
+
+ CHECK_IS_BINARY_UPGRADE;
+
+ /* lock the table the attribute belongs to */
+ tablerel = heap_open(table_id, AccessExclusiveLock);
+
+ /* Lock the attribute row and get the data */
+ attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
+ atttup = SearchSysCacheAttName(table_id, cattname);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %s of relation %u",
+ cattname, table_id);
+ attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /* get an array value from the value string */
+
+ /* find the io info for an arbitrary array type to get array_in Oid */
+ missingval = OidFunctionCall3(
+ F_ARRAY_IN,
+ CStringGetDatum(cvalue),
+ ObjectIdGetDatum(attStruct->atttypid),
+ Int32GetDatum(attStruct->atttypmod));
+
+ /* update the tuple - set atthasmissing and attmissingval */
+ MemSet(valuesAtt, 0, sizeof(valuesAtt));
+ MemSet(nullsAtt, false, sizeof(nullsAtt));
+ MemSet(replacesAtt, false, sizeof(replacesAtt));
+
+ valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
+ replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
+ replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
+
+ newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
+ valuesAtt, nullsAtt, replacesAtt);
+ CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
+
+ /* clean up */
+ ReleaseSysCache(atttup);
+ heap_close(attrrel, RowExclusiveLock);
+ heap_close(tablerel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea2f022..22be3c2 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8095,6 +8095,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_typstorage;
int i_attnotnull;
int i_atthasdef;
+ int i_atthasmissing;
int i_attidentity;
int i_attisdropped;
int i_attlen;
@@ -8103,6 +8104,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_attoptions;
int i_attcollation;
int i_attfdwoptions;
+ int i_attmissingval;
PGresult *res;
int ntups;
bool hasdefaults;
@@ -8132,7 +8134,33 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
resetPQExpBuffer(q);
- if (fout->remoteVersion >= 100000)
+ if (fout->remoteVersion >= 110000)
+ {
+ /* atthasmissing and attmissingval are new in 11 */
+ appendPQExpBuffer(q, "SELECT a.attnum, a.attname, a.atttypmod, "
+ "a.attstattarget, a.attstorage, t.typstorage, "
+ "a.attnotnull, a.atthasdef, a.attisdropped, "
+ "a.attlen, a.attalign, a.attislocal, "
+ "pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
+ "array_to_string(a.attoptions, ', ') AS attoptions, "
+ "CASE WHEN a.attcollation <> t.typcollation "
+ "THEN a.attcollation ELSE 0 END AS attcollation, "
+ "a.atthasmissing, a.attidentity, "
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(attfdwoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attfdwoptions ,"
+ "a.attmissingval "
+ "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
+ "ON a.atttypid = t.oid "
+ "WHERE a.attrelid = '%u'::pg_catalog.oid "
+ "AND a.attnum > 0::pg_catalog.int2 "
+ "ORDER BY a.attnum",
+ tbinfo->dobj.catId.oid);
+ }
+ else if (fout->remoteVersion >= 100000)
{
/*
* attidentity is new in version 10.
@@ -8258,6 +8286,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_typstorage = PQfnumber(res, "typstorage");
i_attnotnull = PQfnumber(res, "attnotnull");
i_atthasdef = PQfnumber(res, "atthasdef");
+ i_atthasmissing = PQfnumber(res, "atthasmissing");
i_attidentity = PQfnumber(res, "attidentity");
i_attisdropped = PQfnumber(res, "attisdropped");
i_attlen = PQfnumber(res, "attlen");
@@ -8266,6 +8295,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_attoptions = PQfnumber(res, "attoptions");
i_attcollation = PQfnumber(res, "attcollation");
i_attfdwoptions = PQfnumber(res, "attfdwoptions");
+ i_attmissingval = PQfnumber(res, "attmissingval");
tbinfo->numatts = ntups;
tbinfo->attnames = (char **) pg_malloc(ntups * sizeof(char *));
@@ -8274,6 +8304,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget = (int *) pg_malloc(ntups * sizeof(int));
tbinfo->attstorage = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->typstorage = (char *) pg_malloc(ntups * sizeof(char));
+ tbinfo->atthasmissing = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attidentity = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->attisdropped = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attlen = (int *) pg_malloc(ntups * sizeof(int));
@@ -8282,6 +8313,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8299,6 +8331,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget[j] = atoi(PQgetvalue(res, j, i_attstattarget));
tbinfo->attstorage[j] = *(PQgetvalue(res, j, i_attstorage));
tbinfo->typstorage[j] = *(PQgetvalue(res, j, i_typstorage));
+ tbinfo->atthasmissing[j] = (i_atthasmissing >= 0 ? (PQgetvalue(res, j, i_atthasmissing)[0] == 't') : false);
tbinfo->attidentity[j] = (i_attidentity >= 0 ? *(PQgetvalue(res, j, i_attidentity)) : '\0');
tbinfo->needs_override = tbinfo->needs_override || (tbinfo->attidentity[j] == ATTRIBUTE_IDENTITY_ALWAYS);
tbinfo->attisdropped[j] = (PQgetvalue(res, j, i_attisdropped)[0] == 't');
@@ -8309,6 +8342,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, j, i_attoptions));
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, i_attcollation));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, i_attfdwoptions));
+ tbinfo->attmissingval[j] = (tbinfo->atthasmissing[j] ? pg_strdup(PQgetvalue(res, j, i_attmissingval)) : NULL );
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, i_atthasdef)[0] == 't')
hasdefaults = true;
@@ -15659,6 +15693,29 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
appendPQExpBufferStr(q, ";\n");
/*
+ * in binary upgrade mode, update the catalog with any missing values
+ * that might be present.
+ */
+ if (dopt->binary_upgrade)
+ {
+ for (j = 0; j < tbinfo->numatts; j++)
+ {
+ if (tbinfo->atthasmissing[j])
+ {
+ appendPQExpBufferStr(q, "\n-- set missing value.\n");
+ appendPQExpBufferStr(q,
+ "SELECT pg_catalog.binary_upgrade_set_missing_value(");
+ appendStringLiteralAH(q,qualrelname, fout);
+ appendPQExpBufferStr(q, "::pg_catalog.regclass,");
+ appendStringLiteralAH(q, tbinfo->attnames[j], fout);
+ appendPQExpBufferStr(q,",");
+ appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
+ appendPQExpBufferStr(q,");\n\n");
+ }
+ }
+ }
+
+ /*
* To create binary-compatible heap files, we have to ensure the same
* physical column order, including dropped columns, as in the
* original. Therefore, we create dropped columns above and drop them
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e96c662..c1681b3 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -309,6 +309,7 @@ typedef struct _tableInfo
char *attstorage; /* attribute storage scheme */
char *typstorage; /* type storage scheme */
bool *attisdropped; /* true if attr is dropped; don't dump it */
+ bool *atthasmissing; /* true if the attribute has a missing value */
char *attidentity;
int *attlen; /* attribute length, used by binary_upgrade */
char *attalign; /* attribute align, used by binary_upgrade */
@@ -316,6 +317,7 @@ typedef struct _tableInfo
char **attoptions; /* per-attribute options */
Oid *attcollation; /* per-attribute collation selection */
char **attfdwoptions; /* per-attribute fdw options */
+ char **attmissingval; /* per attribute missing value */
bool *notnull; /* NOT NULL constraints on attributes */
bool *inhNotNull; /* true if NOT NULL is inherited */
struct _attrDefInfo **attrdefs; /* DEFAULT expressions */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66c6c22..9834e38 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10037,6 +10037,10 @@
proname => 'binary_upgrade_set_record_init_privs', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'bool',
prosrc => 'binary_upgrade_set_record_init_privs' },
+{ oid => '4101', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_missing_value', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid text text',
+ prosrc => 'binary_upgrade_set_missing_value' },
# replication/origin.h
{ oid => '6003', descr => 'create a replication origin',
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 01:04 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-21 01:04 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
Hi,
On 2018-06-20 20:53:34 -0400, Andrew Dunstan wrote:
> This version adds a lock on the table owning the attribute.
Cool.
>
> /*
> + * in binary upgrade mode, update the catalog with any missing values
> + * that might be present.
> + */
> + if (dopt->binary_upgrade)
> + {
> + for (j = 0; j < tbinfo->numatts; j++)
> + {
> + if (tbinfo->atthasmissing[j])
> + {
> + appendPQExpBufferStr(q, "\n-- set missing value.\n");
> + appendPQExpBufferStr(q,
> + "SELECT pg_catalog.binary_upgrade_set_missing_value(");
> + appendStringLiteralAH(q,qualrelname, fout);
missing space. Probably couldn't hurt to run the changed files through
pgindent and fix the damage...
> + appendPQExpBufferStr(q, "::pg_catalog.regclass,");
> + appendStringLiteralAH(q, tbinfo->attnames[j], fout);
> + appendPQExpBufferStr(q,",");
same.
> + appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
> + appendPQExpBufferStr(q,");\n\n");
same.
Looks reasonable to me, but I've not tested it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 12:07 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 12:07 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
On 06/20/2018 09:04 PM, Andres Freund wrote:
> Probably couldn't hurt to run the changed files through
> pgindent and fix the damage...
>
>
> Looks reasonable to me, but I've not tested it.
Thanks
Patch after pgindent attached. This will involve a catversion bump since
we're adding a new function.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-patch] fix-default-6.patch (11.2K, ../../[email protected]/2-fix-default-6.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 0c54b02..7a2f785 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,13 +11,20 @@
#include "postgres.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
#include "catalog/binary_upgrade.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/extension.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -192,3 +199,63 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+Datum
+binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
+{
+ Oid table_id = PG_GETARG_OID(0);
+ text *attname = PG_GETARG_TEXT_P(1);
+ text *value = PG_GETARG_TEXT_P(2);
+ Datum valuesAtt[Natts_pg_attribute];
+ bool nullsAtt[Natts_pg_attribute];
+ bool replacesAtt[Natts_pg_attribute];
+ Datum missingval;
+ Form_pg_attribute attStruct;
+ Relation attrrel,
+ tablerel;
+ HeapTuple atttup,
+ newtup;
+ char *cattname = text_to_cstring(attname);
+ char *cvalue = text_to_cstring(value);
+
+ CHECK_IS_BINARY_UPGRADE;
+
+ /* lock the table the attribute belongs to */
+ tablerel = heap_open(table_id, AccessExclusiveLock);
+
+ /* Lock the attribute row and get the data */
+ attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
+ atttup = SearchSysCacheAttName(table_id, cattname);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %s of relation %u",
+ cattname, table_id);
+ attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /* get an array value from the value string */
+ missingval = OidFunctionCall3(
+ F_ARRAY_IN,
+ CStringGetDatum(cvalue),
+ ObjectIdGetDatum(attStruct->atttypid),
+ Int32GetDatum(attStruct->atttypmod));
+
+ /* update the tuple - set atthasmissing and attmissingval */
+ MemSet(valuesAtt, 0, sizeof(valuesAtt));
+ MemSet(nullsAtt, false, sizeof(nullsAtt));
+ MemSet(replacesAtt, false, sizeof(replacesAtt));
+
+ valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
+ replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
+ replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
+
+ newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
+ valuesAtt, nullsAtt, replacesAtt);
+ CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
+
+ /* clean up */
+ ReleaseSysCache(atttup);
+ heap_close(attrrel, RowExclusiveLock);
+ heap_close(tablerel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea2f022..e06ed60 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8095,6 +8095,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_typstorage;
int i_attnotnull;
int i_atthasdef;
+ int i_atthasmissing;
int i_attidentity;
int i_attisdropped;
int i_attlen;
@@ -8103,6 +8104,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_attoptions;
int i_attcollation;
int i_attfdwoptions;
+ int i_attmissingval;
PGresult *res;
int ntups;
bool hasdefaults;
@@ -8132,7 +8134,33 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
resetPQExpBuffer(q);
- if (fout->remoteVersion >= 100000)
+ if (fout->remoteVersion >= 110000)
+ {
+ /* atthasmissing and attmissingval are new in 11 */
+ appendPQExpBuffer(q, "SELECT a.attnum, a.attname, a.atttypmod, "
+ "a.attstattarget, a.attstorage, t.typstorage, "
+ "a.attnotnull, a.atthasdef, a.attisdropped, "
+ "a.attlen, a.attalign, a.attislocal, "
+ "pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
+ "array_to_string(a.attoptions, ', ') AS attoptions, "
+ "CASE WHEN a.attcollation <> t.typcollation "
+ "THEN a.attcollation ELSE 0 END AS attcollation, "
+ "a.atthasmissing, a.attidentity, "
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(attfdwoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attfdwoptions ,"
+ "a.attmissingval "
+ "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
+ "ON a.atttypid = t.oid "
+ "WHERE a.attrelid = '%u'::pg_catalog.oid "
+ "AND a.attnum > 0::pg_catalog.int2 "
+ "ORDER BY a.attnum",
+ tbinfo->dobj.catId.oid);
+ }
+ else if (fout->remoteVersion >= 100000)
{
/*
* attidentity is new in version 10.
@@ -8258,6 +8286,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_typstorage = PQfnumber(res, "typstorage");
i_attnotnull = PQfnumber(res, "attnotnull");
i_atthasdef = PQfnumber(res, "atthasdef");
+ i_atthasmissing = PQfnumber(res, "atthasmissing");
i_attidentity = PQfnumber(res, "attidentity");
i_attisdropped = PQfnumber(res, "attisdropped");
i_attlen = PQfnumber(res, "attlen");
@@ -8266,6 +8295,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_attoptions = PQfnumber(res, "attoptions");
i_attcollation = PQfnumber(res, "attcollation");
i_attfdwoptions = PQfnumber(res, "attfdwoptions");
+ i_attmissingval = PQfnumber(res, "attmissingval");
tbinfo->numatts = ntups;
tbinfo->attnames = (char **) pg_malloc(ntups * sizeof(char *));
@@ -8274,6 +8304,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget = (int *) pg_malloc(ntups * sizeof(int));
tbinfo->attstorage = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->typstorage = (char *) pg_malloc(ntups * sizeof(char));
+ tbinfo->atthasmissing = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attidentity = (char *) pg_malloc(ntups * sizeof(char));
tbinfo->attisdropped = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attlen = (int *) pg_malloc(ntups * sizeof(int));
@@ -8282,6 +8313,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8299,6 +8331,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attstattarget[j] = atoi(PQgetvalue(res, j, i_attstattarget));
tbinfo->attstorage[j] = *(PQgetvalue(res, j, i_attstorage));
tbinfo->typstorage[j] = *(PQgetvalue(res, j, i_typstorage));
+ tbinfo->atthasmissing[j] = (i_atthasmissing >= 0 ? (PQgetvalue(res, j, i_atthasmissing)[0] == 't') : false);
tbinfo->attidentity[j] = (i_attidentity >= 0 ? *(PQgetvalue(res, j, i_attidentity)) : '\0');
tbinfo->needs_override = tbinfo->needs_override || (tbinfo->attidentity[j] == ATTRIBUTE_IDENTITY_ALWAYS);
tbinfo->attisdropped[j] = (PQgetvalue(res, j, i_attisdropped)[0] == 't');
@@ -8309,6 +8342,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, j, i_attoptions));
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, i_attcollation));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, i_attfdwoptions));
+ tbinfo->attmissingval[j] = (tbinfo->atthasmissing[j] ? pg_strdup(PQgetvalue(res, j, i_attmissingval)) : NULL);
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, i_atthasdef)[0] == 't')
hasdefaults = true;
@@ -14615,18 +14649,23 @@ dumpDefaultACL(Archive *fout, DefaultACLInfo *daclinfo)
{
case DEFACLOBJ_RELATION:
type = "TABLES";
+
break;
case DEFACLOBJ_SEQUENCE:
type = "SEQUENCES";
+
break;
case DEFACLOBJ_FUNCTION:
type = "FUNCTIONS";
+
break;
case DEFACLOBJ_TYPE:
type = "TYPES";
+
break;
case DEFACLOBJ_NAMESPACE:
type = "SCHEMAS";
+
break;
default:
/* shouldn't get here */
@@ -15659,6 +15698,29 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
appendPQExpBufferStr(q, ";\n");
/*
+ * in binary upgrade mode, update the catalog with any missing values
+ * that might be present.
+ */
+ if (dopt->binary_upgrade)
+ {
+ for (j = 0; j < tbinfo->numatts; j++)
+ {
+ if (tbinfo->atthasmissing[j])
+ {
+ appendPQExpBufferStr(q, "\n-- set missing value.\n");
+ appendPQExpBufferStr(q,
+ "SELECT pg_catalog.binary_upgrade_set_missing_value(");
+ appendStringLiteralAH(q, qualrelname, fout);
+ appendPQExpBufferStr(q, "::pg_catalog.regclass,");
+ appendStringLiteralAH(q, tbinfo->attnames[j], fout);
+ appendPQExpBufferStr(q, ",");
+ appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
+ appendPQExpBufferStr(q, ");\n\n");
+ }
+ }
+ }
+
+ /*
* To create binary-compatible heap files, we have to ensure the same
* physical column order, including dropped columns, as in the
* original. Therefore, we create dropped columns above and drop them
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e96c662..c1681b3 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -309,6 +309,7 @@ typedef struct _tableInfo
char *attstorage; /* attribute storage scheme */
char *typstorage; /* type storage scheme */
bool *attisdropped; /* true if attr is dropped; don't dump it */
+ bool *atthasmissing; /* true if the attribute has a missing value */
char *attidentity;
int *attlen; /* attribute length, used by binary_upgrade */
char *attalign; /* attribute align, used by binary_upgrade */
@@ -316,6 +317,7 @@ typedef struct _tableInfo
char **attoptions; /* per-attribute options */
Oid *attcollation; /* per-attribute collation selection */
char **attfdwoptions; /* per-attribute fdw options */
+ char **attmissingval; /* per attribute missing value */
bool *notnull; /* NOT NULL constraints on attributes */
bool *inhNotNull; /* true if NOT NULL is inherited */
struct _attrDefInfo **attrdefs; /* DEFAULT expressions */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66c6c22..9834e38 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10037,6 +10037,10 @@
proname => 'binary_upgrade_set_record_init_privs', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'bool',
prosrc => 'binary_upgrade_set_record_init_privs' },
+{ oid => '4101', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_missing_value', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid text text',
+ prosrc => 'binary_upgrade_set_missing_value' },
# replication/origin.h
{ oid => '6003', descr => 'create a replication origin',
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 16:04 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 2 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-21 16:04 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> Patch after pgindent attached. This will involve a catversion bump since
> we're adding a new function.
This isn't really ready to go. One clear problem is that you broke
pg_dump'ing from any pre-v11 version, because you did not add suitable
null outputs to the pre-v11 query variants in getTableAttrs.
Personally, I'd have made the pg_dump changes simpler and cheaper by
storing only one new value per column not two. You could have just
stored attmissingval, or if you want to be paranoid you could do
case when atthasmissing then attmissingval else null end
We could do without the unrelated whitespace changes in dumpDefaultACL,
too (or did pgindent introduce those? Seems surprising ... oh, looks
like "type" has somehow snuck into typedefs.list. Time for another
blacklist entry.)
In dumpTableSchema, I'd suggest emitting the numeric table OID, rather
than hacking around with regclass.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 16:08 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-21 16:08 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
I wrote:
> We could do without the unrelated whitespace changes in dumpDefaultACL,
> too (or did pgindent introduce those? Seems surprising ... oh, looks
> like "type" has somehow snuck into typedefs.list. Time for another
> blacklist entry.)
Hmm .. actually, "type" already is in pgindent's blacklist. Are you
using an up-to-date pgindent'ing setup?
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 16:30 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-21 16:30 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; +Cc: [email protected]
On June 21, 2018 9:04:28 AM PDT, Tom Lane <[email protected]> wrote:
>Andrew Dunstan <[email protected]> writes:
>> Patch after pgindent attached. This will involve a catversion bump
>since
>> we're adding a new function.
>
>This isn't really ready to go. One clear problem is that you broke
>pg_dump'ing from any pre-v11 version, because you did not add suitable
>null outputs to the pre-v11 query variants in getTableAttrs.
Thought the same for a bit - think it's OK though, because there a check for PQfnumber being bigger than 0...
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 16:39 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-21 16:39 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
Andres Freund <[email protected]> writes:
> On June 21, 2018 9:04:28 AM PDT, Tom Lane <[email protected]> wrote:
>> This isn't really ready to go. One clear problem is that you broke
>> pg_dump'ing from any pre-v11 version, because you did not add suitable
>> null outputs to the pre-v11 query variants in getTableAttrs.
> Thought the same for a bit - think it's OK though, because there a check for PQfnumber being bigger than 0...
It won't actually crash, but it will spit out lots of ugly warnings.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:02 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 17:02 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Andres Freund <[email protected]>; +Cc: [email protected]
On 06/21/2018 12:39 PM, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
>> On June 21, 2018 9:04:28 AM PDT, Tom Lane <[email protected]> wrote:
>>> This isn't really ready to go. One clear problem is that you broke
>>> pg_dump'ing from any pre-v11 version, because you did not add suitable
>>> null outputs to the pre-v11 query variants in getTableAttrs.
>> Thought the same for a bit - think it's OK though, because there a check for PQfnumber being bigger than 0...
> It won't actually crash, but it will spit out lots of ugly warnings.
>
>
I followed the pattern used for attidentity. But why will it spit out
warnings? It doesn't ask for the missing stuff from an older version.
Incidentally, I just checked this by applying the patch and then running
through the buildfarm's cross version upgrade check. All was kosher.
Upgrade from all live branches to the patched master went without a hitch.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:18 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-21 17:18 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> On 06/21/2018 12:39 PM, Tom Lane wrote:
>> Andres Freund <[email protected]> writes:
> On June 21, 2018 9:04:28 AM PDT, Tom Lane <[email protected]> wrote:
> This isn't really ready to go. One clear problem is that you broke
> pg_dump'ing from any pre-v11 version, because you did not add suitable
> null outputs to the pre-v11 query variants in getTableAttrs.
> I followed the pattern used for attidentity. But why will it spit out
> warnings? It doesn't ask for the missing stuff from an older version.
Oh, I see. Well, the short answer is that that's not the style we use
in pg_dump, and the attidentity code is inappropriate/wrong too IMO.
When something has been done one way a hundred times before, thinking
you're too smart for that and you'll do it some other way does not lend
itself to code clarity or reviewability.
I might be OK with a patch that converts *all* of pg_dump's cross-version
difference handling code to depend on PQfnumber silently returning -1
rather than failing, but I don't want to see it done like that in just
one or two places.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:28 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 17:28 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
On 06/21/2018 01:18 PM, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
>> On 06/21/2018 12:39 PM, Tom Lane wrote:
>>> Andres Freund <[email protected]> writes:
>> On June 21, 2018 9:04:28 AM PDT, Tom Lane <[email protected]> wrote:
>> This isn't really ready to go. One clear problem is that you broke
>> pg_dump'ing from any pre-v11 version, because you did not add suitable
>> null outputs to the pre-v11 query variants in getTableAttrs.
>> I followed the pattern used for attidentity. But why will it spit out
>> warnings? It doesn't ask for the missing stuff from an older version.
> Oh, I see. Well, the short answer is that that's not the style we use
> in pg_dump, and the attidentity code is inappropriate/wrong too IMO.
> When something has been done one way a hundred times before, thinking
> you're too smart for that and you'll do it some other way does not lend
> itself to code clarity or reviewability.
>
> I might be OK with a patch that converts *all* of pg_dump's cross-version
> difference handling code to depend on PQfnumber silently returning -1
> rather than failing, but I don't want to see it done like that in just
> one or two places.
>
>
I don't mind changing it. But please note that I wouldn't have done it
that way unless there were a precedent. I fully expected to add dummy
values to all the previous queries, but when I couldn't find attidentity
in them to put them next to I followed that example.
I don't think it's at all a bad idea, TBH. It means you only have to add
one query per version rather than having to adjust all of them.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:31 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 17:31 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
On 06/21/2018 12:08 PM, Tom Lane wrote:
> I wrote:
>> We could do without the unrelated whitespace changes in dumpDefaultACL,
>> too (or did pgindent introduce those? Seems surprising ... oh, looks
>> like "type" has somehow snuck into typedefs.list. Time for another
>> blacklist entry.)
> Hmm .. actually, "type" already is in pgindent's blacklist. Are you
> using an up-to-date pgindent'ing setup?
>
>
I am now, This was a holdover from before I updated.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:40 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 0 replies; 57+ messages in thread
From: Andres Freund @ 2018-06-21 17:40 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; [email protected]
Hi,
On 2018-06-21 13:28:46 -0400, Andrew Dunstan wrote:
> I don't think it's at all a bad idea, TBH. It means you only have to add one
> query per version rather than having to adjust all of them.
I'd even say it's an excellent idea.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:44 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 2 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-21 17:44 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> On 06/21/2018 01:18 PM, Tom Lane wrote:
>> I might be OK with a patch that converts *all* of pg_dump's cross-version
>> difference handling code to depend on PQfnumber silently returning -1
>> rather than failing, but I don't want to see it done like that in just
>> one or two places.
> I don't mind changing it. But please note that I wouldn't have done it
> that way unless there were a precedent. I fully expected to add dummy
> values to all the previous queries, but when I couldn't find attidentity
> in them to put them next to I followed that example.
Actually, now that I think about it, there is a concrete reason for the
historical pattern: it provides a cross-check that you did not fat-finger
the query, ie misspell the column alias vs the PQfnumber parameter. This
gets more valuable the more per-version variants of the query there are.
With the way the attidentity code does it, it would just silently act as
though the column has its default value, which you might or might not
notice in cursory testing. Getting visible bleats about column number -1
is much more likely to get your attention.
So I'm thinking that the attidentity code is just wrong, and you should
change that too while you're at it.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:46 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Andres Freund @ 2018-06-21 17:46 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
On 2018-06-21 13:44:19 -0400, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
> > On 06/21/2018 01:18 PM, Tom Lane wrote:
> >> I might be OK with a patch that converts *all* of pg_dump's cross-version
> >> difference handling code to depend on PQfnumber silently returning -1
> >> rather than failing, but I don't want to see it done like that in just
> >> one or two places.
>
> > I don't mind changing it. But please note that I wouldn't have done it
> > that way unless there were a precedent. I fully expected to add dummy
> > values to all the previous queries, but when I couldn't find attidentity
> > in them to put them next to I followed that example.
>
> Actually, now that I think about it, there is a concrete reason for the
> historical pattern: it provides a cross-check that you did not fat-finger
> the query, ie misspell the column alias vs the PQfnumber parameter. This
> gets more valuable the more per-version variants of the query there are.
> With the way the attidentity code does it, it would just silently act as
> though the column has its default value, which you might or might not
> notice in cursory testing. Getting visible bleats about column number -1
> is much more likely to get your attention.
To me that benefit is far smaller than the increased likelihood of
screwing something up because you'd to touch pg_dump support for
postgres versions one likely doesn't readily have available for testing.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:49 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 17:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
On 06/21/2018 01:44 PM, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
>> On 06/21/2018 01:18 PM, Tom Lane wrote:
>>> I might be OK with a patch that converts *all* of pg_dump's cross-version
>>> difference handling code to depend on PQfnumber silently returning -1
>>> rather than failing, but I don't want to see it done like that in just
>>> one or two places.
>> I don't mind changing it. But please note that I wouldn't have done it
>> that way unless there were a precedent. I fully expected to add dummy
>> values to all the previous queries, but when I couldn't find attidentity
>> in them to put them next to I followed that example.
> Actually, now that I think about it, there is a concrete reason for the
> historical pattern: it provides a cross-check that you did not fat-finger
> the query, ie misspell the column alias vs the PQfnumber parameter. This
> gets more valuable the more per-version variants of the query there are.
> With the way the attidentity code does it, it would just silently act as
> though the column has its default value, which you might or might not
> notice in cursory testing. Getting visible bleats about column number -1
> is much more likely to get your attention.
>
> So I'm thinking that the attidentity code is just wrong, and you should
> change that too while you're at it.
>
>
That should be backpatched if changed, no? I don't think we'd want this
to get out of sync between the branches. It would make later
backpatching more difficult for one thing.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:50 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-21 17:50 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; [email protected]
Andres Freund <[email protected]> writes:
> On 2018-06-21 13:44:19 -0400, Tom Lane wrote:
>> Actually, now that I think about it, there is a concrete reason for the
>> historical pattern: it provides a cross-check that you did not fat-finger
>> the query, ie misspell the column alias vs the PQfnumber parameter. This
>> gets more valuable the more per-version variants of the query there are.
>> With the way the attidentity code does it, it would just silently act as
>> though the column has its default value, which you might or might not
>> notice in cursory testing. Getting visible bleats about column number -1
>> is much more likely to get your attention.
> To me that benefit is far smaller than the increased likelihood of
> screwing something up because you'd to touch pg_dump support for
> postgres versions one likely doesn't readily have available for testing.
I beg to differ: code that works "by omission" is just as likely to be
wrong. Anyway, our solution for unmaintainable back branches has been
to drop support, cf where we got rid of the pre-8.0 queries awhile back.
One (admittedly small) problem with the "i_attidentity >= 0 ? ..." coding
is that it's not obvious when you can get rid of it because you dropped
the last old branch that needed it. After that, it's just a hazard for
hiding mistakes.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 17:53 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-21 17:53 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> On 06/21/2018 01:44 PM, Tom Lane wrote:
>> So I'm thinking that the attidentity code is just wrong, and you should
>> change that too while you're at it.
> That should be backpatched if changed, no? I don't think we'd want this
> to get out of sync between the branches. It would make later
> backpatching more difficult for one thing.
If you feel like it. But if there's attmissingval code right next to it
as of v11, then backpatches wouldn't apply cleanly anyway due to lack of
context match, so I doubt there's really much gain to be had.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 18:51 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-21 18:51 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
On 06/21/2018 01:53 PM, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
>> On 06/21/2018 01:44 PM, Tom Lane wrote:
>>> So I'm thinking that the attidentity code is just wrong, and you should
>>> change that too while you're at it.
>> That should be backpatched if changed, no? I don't think we'd want this
>> to get out of sync between the branches. It would make later
>> backpatching more difficult for one thing.
> If you feel like it. But if there's attmissingval code right next to it
> as of v11, then backpatches wouldn't apply cleanly anyway due to lack of
> context match, so I doubt there's really much gain to be had.
>
>
I left that for a separate exercise. I think this does things the way
you want. I must admit to being a bit surprised, I was expecting you to
have more to say about the upgrade function than the pg_dump changes :-)
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-patch] fix-default-8.patch (11.1K, ../../[email protected]/2-fix-default-8.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 0c54b02..7a2f785 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,13 +11,20 @@
#include "postgres.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
#include "catalog/binary_upgrade.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/extension.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -192,3 +199,63 @@ binary_upgrade_set_record_init_privs(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+Datum
+binary_upgrade_set_missing_value(PG_FUNCTION_ARGS)
+{
+ Oid table_id = PG_GETARG_OID(0);
+ text *attname = PG_GETARG_TEXT_P(1);
+ text *value = PG_GETARG_TEXT_P(2);
+ Datum valuesAtt[Natts_pg_attribute];
+ bool nullsAtt[Natts_pg_attribute];
+ bool replacesAtt[Natts_pg_attribute];
+ Datum missingval;
+ Form_pg_attribute attStruct;
+ Relation attrrel,
+ tablerel;
+ HeapTuple atttup,
+ newtup;
+ char *cattname = text_to_cstring(attname);
+ char *cvalue = text_to_cstring(value);
+
+ CHECK_IS_BINARY_UPGRADE;
+
+ /* lock the table the attribute belongs to */
+ tablerel = heap_open(table_id, AccessExclusiveLock);
+
+ /* Lock the attribute row and get the data */
+ attrrel = heap_open(AttributeRelationId, RowExclusiveLock);
+ atttup = SearchSysCacheAttName(table_id, cattname);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %s of relation %u",
+ cattname, table_id);
+ attStruct = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /* get an array value from the value string */
+ missingval = OidFunctionCall3(
+ F_ARRAY_IN,
+ CStringGetDatum(cvalue),
+ ObjectIdGetDatum(attStruct->atttypid),
+ Int32GetDatum(attStruct->atttypmod));
+
+ /* update the tuple - set atthasmissing and attmissingval */
+ MemSet(valuesAtt, 0, sizeof(valuesAtt));
+ MemSet(nullsAtt, false, sizeof(nullsAtt));
+ MemSet(replacesAtt, false, sizeof(replacesAtt));
+
+ valuesAtt[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(true);
+ replacesAtt[Anum_pg_attribute_atthasmissing - 1] = true;
+ valuesAtt[Anum_pg_attribute_attmissingval - 1] = missingval;
+ replacesAtt[Anum_pg_attribute_attmissingval - 1] = true;
+
+ newtup = heap_modify_tuple(atttup, RelationGetDescr(attrrel),
+ valuesAtt, nullsAtt, replacesAtt);
+ CatalogTupleUpdate(attrrel, &newtup->t_self, newtup);
+
+ /* clean up */
+ ReleaseSysCache(atttup);
+ heap_close(attrrel, RowExclusiveLock);
+ heap_close(tablerel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea2f022..40d1eb0 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8103,6 +8103,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
int i_attoptions;
int i_attcollation;
int i_attfdwoptions;
+ int i_attmissingval;
PGresult *res;
int ntups;
bool hasdefaults;
@@ -8132,7 +8133,34 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
resetPQExpBuffer(q);
- if (fout->remoteVersion >= 100000)
+ if (fout->remoteVersion >= 110000)
+ {
+ /* atthasmissing and attmissingval are new in 11 */
+ appendPQExpBuffer(q, "SELECT a.attnum, a.attname, a.atttypmod, "
+ "a.attstattarget, a.attstorage, t.typstorage, "
+ "a.attnotnull, a.atthasdef, a.attisdropped, "
+ "a.attlen, a.attalign, a.attislocal, "
+ "pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
+ "array_to_string(a.attoptions, ', ') AS attoptions, "
+ "CASE WHEN a.attcollation <> t.typcollation "
+ "THEN a.attcollation ELSE 0 END AS attcollation, "
+ "a.attidentity, "
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(attfdwoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attfdwoptions ,"
+ "CASE WHEN a.atthasmissing THEN a.attmissingval "
+ "ELSE null END AS attmissingval "
+ "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
+ "ON a.atttypid = t.oid "
+ "WHERE a.attrelid = '%u'::pg_catalog.oid "
+ "AND a.attnum > 0::pg_catalog.int2 "
+ "ORDER BY a.attnum",
+ tbinfo->dobj.catId.oid);
+ }
+ else if (fout->remoteVersion >= 100000)
{
/*
* attidentity is new in version 10.
@@ -8151,7 +8179,8 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"' ' || pg_catalog.quote_literal(option_value) "
"FROM pg_catalog.pg_options_to_table(attfdwoptions) "
"ORDER BY option_name"
- "), E',\n ') AS attfdwoptions "
+ "), E',\n ') AS attfdwoptions ,"
+ "NULL as attmissingval "
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
"ON a.atttypid = t.oid "
"WHERE a.attrelid = '%u'::pg_catalog.oid "
@@ -8177,7 +8206,8 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"' ' || pg_catalog.quote_literal(option_value) "
"FROM pg_catalog.pg_options_to_table(attfdwoptions) "
"ORDER BY option_name"
- "), E',\n ') AS attfdwoptions "
+ "), E',\n ') AS attfdwoptions, "
+ "NULL as attmissingval "
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
"ON a.atttypid = t.oid "
"WHERE a.attrelid = '%u'::pg_catalog.oid "
@@ -8201,7 +8231,8 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"array_to_string(a.attoptions, ', ') AS attoptions, "
"CASE WHEN a.attcollation <> t.typcollation "
"THEN a.attcollation ELSE 0 END AS attcollation, "
- "NULL AS attfdwoptions "
+ "NULL AS attfdwoptions, "
+ "NULL as attmissingval "
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
"ON a.atttypid = t.oid "
"WHERE a.attrelid = '%u'::pg_catalog.oid "
@@ -8219,7 +8250,8 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
"array_to_string(a.attoptions, ', ') AS attoptions, "
"0 AS attcollation, "
- "NULL AS attfdwoptions "
+ "NULL AS attfdwoptions, "
+ "NULL as attmissingval "
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
"ON a.atttypid = t.oid "
"WHERE a.attrelid = '%u'::pg_catalog.oid "
@@ -8236,7 +8268,8 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"a.attlen, a.attalign, a.attislocal, "
"pg_catalog.format_type(t.oid,a.atttypmod) AS atttypname, "
"'' AS attoptions, 0 AS attcollation, "
- "NULL AS attfdwoptions "
+ "NULL AS attfdwoptions, "
+ "NULL as attmissingval "
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
"ON a.atttypid = t.oid "
"WHERE a.attrelid = '%u'::pg_catalog.oid "
@@ -8266,6 +8299,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
i_attoptions = PQfnumber(res, "attoptions");
i_attcollation = PQfnumber(res, "attcollation");
i_attfdwoptions = PQfnumber(res, "attfdwoptions");
+ i_attmissingval = PQfnumber(res, "attmissingval");
tbinfo->numatts = ntups;
tbinfo->attnames = (char **) pg_malloc(ntups * sizeof(char *));
@@ -8282,6 +8316,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8309,6 +8344,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, j, i_attoptions));
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, i_attcollation));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, i_attfdwoptions));
+ tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, i_attmissingval));
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, i_atthasdef)[0] == 't')
hasdefaults = true;
@@ -15659,6 +15695,29 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
appendPQExpBufferStr(q, ";\n");
/*
+ * in binary upgrade mode, update the catalog with any missing values
+ * that might be present.
+ */
+ if (dopt->binary_upgrade)
+ {
+ for (j = 0; j < tbinfo->numatts; j++)
+ {
+ if (tbinfo->attmissingval[j][0] != '\0')
+ {
+ appendPQExpBufferStr(q, "\n-- set missing value.\n");
+ appendPQExpBufferStr(q,
+ "SELECT pg_catalog.binary_upgrade_set_missing_value(");
+ appendStringLiteralAH(q, qualrelname, fout);
+ appendPQExpBufferStr(q, "::pg_catalog.regclass,");
+ appendStringLiteralAH(q, tbinfo->attnames[j], fout);
+ appendPQExpBufferStr(q, ",");
+ appendStringLiteralAH(q, tbinfo->attmissingval[j], fout);
+ appendPQExpBufferStr(q, ");\n\n");
+ }
+ }
+ }
+
+ /*
* To create binary-compatible heap files, we have to ensure the same
* physical column order, including dropped columns, as in the
* original. Therefore, we create dropped columns above and drop them
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e96c662..b904379 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -316,6 +316,7 @@ typedef struct _tableInfo
char **attoptions; /* per-attribute options */
Oid *attcollation; /* per-attribute collation selection */
char **attfdwoptions; /* per-attribute fdw options */
+ char **attmissingval; /* per attribute missing value */
bool *notnull; /* NOT NULL constraints on attributes */
bool *inhNotNull; /* true if NOT NULL is inherited */
struct _attrDefInfo **attrdefs; /* DEFAULT expressions */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66c6c22..9834e38 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10037,6 +10037,10 @@
proname => 'binary_upgrade_set_record_init_privs', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'bool',
prosrc => 'binary_upgrade_set_record_init_privs' },
+{ oid => '4101', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_missing_value', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid text text',
+ prosrc => 'binary_upgrade_set_missing_value' },
# replication/origin.h
{ oid => '6003', descr => 'create a replication origin',
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 20:41 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Tom Lane @ 2018-06-21 20:41 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> I left that for a separate exercise. I think this does things the way
> you want. I must admit to being a bit surprised, I was expecting you to
> have more to say about the upgrade function than the pg_dump changes :-)
Well, the upgrade function is certainly a bit ugly (I'm generally allergic
to patches that result in a large increase in the number of #includes in
a file --- it suggests that the code was put in an inappropriate place).
But I don't see a good way to make it better, at least not without heavy
refactoring of StoreAttrDefault so that some code could be shared.
I think this is probably OK now, except for one thing: you're likely
to have issues if a dropped column has an attmissingval, because often
the value won't agree with the bogus "integer" type we use for those;
worse, the array might refer to a type that no longer exists.
One way to improve that is to change
"CASE WHEN a.atthasmissing THEN a.attmissingval "
"ELSE null END AS attmissingval "
to
"CASE WHEN a.atthasmissing AND NOT a.attisdropped THEN a.attmissingval "
"ELSE null END AS attmissingval "
However, this might be another reason to reconsider the decision to use
anyarray: it could easily lead to situations where harmless-looking
queries like "select * from pg_attribute" fail. Or maybe we should tweak
DROP COLUMN so that it forcibly resets atthasmissing/attmissingval along
with setting atthasdropped, so that the case can't arise.
Like Andres, I haven't actually tested, just eyeballed it.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 21:10 Alvaro Herrera <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 57+ messages in thread
From: Alvaro Herrera @ 2018-06-21 21:10 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; [email protected]
In terms of pgindent, I'm surprised about these lines:
+ missingval = OidFunctionCall3(
+ F_ARRAY_IN,
Why did you put a newline there? In ancient times there was a reason
for that in some cases, because pgindent would move the argument to the
left of the open parens, but it doesn't do that anymore and IMO it's
just ugly. We have quite a few leftovers from this ancient practice,
I've been thinking about removing these ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-21 21:20 Tom Lane <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Tom Lane @ 2018-06-21 21:20 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Andres Freund <[email protected]>; [email protected]
Alvaro Herrera <[email protected]> writes:
> In terms of pgindent, I'm surprised about these lines:
> + missingval = OidFunctionCall3(
> + F_ARRAY_IN,
> Why did you put a newline there? In ancient times there was a reason
> for that in some cases, because pgindent would move the argument to the
> left of the open parens, but it doesn't do that anymore and IMO it's
> just ugly. We have quite a few leftovers from this ancient practice,
> I've been thinking about removing these ...
I think some people feel this is good style, but I agree with you
about not liking it. A related practice I could do without is eating
an extra line for an argument-closing paren, as in this example in
tsquery_op.c:
Datum
tsq_mcontained(PG_FUNCTION_ARGS)
{
PG_RETURN_DATUM(
DirectFunctionCall2(
tsq_mcontains,
PG_GETARG_DATUM(1),
PG_GETARG_DATUM(0)
)
);
}
Aside from the waste of vertical space, it's never very clear to me
(nor, evidently, to pgindent) how such a paren ought to be indented.
So to my eye this could be four lines shorter and look better.
regards, tom lane
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Fast default stuff versus pg_upgrade
@ 2018-06-22 13:01 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Andrew Dunstan @ 2018-06-22 13:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; [email protected]
On 06/21/2018 04:41 PM, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
>> I left that for a separate exercise. I think this does things the way
>> you want. I must admit to being a bit surprised, I was expecting you to
>> have more to say about the upgrade function than the pg_dump changes :-)
> Well, the upgrade function is certainly a bit ugly (I'm generally allergic
> to patches that result in a large increase in the number of #includes in
> a file --- it suggests that the code was put in an inappropriate place).
> But I don't see a good way to make it better, at least not without heavy
> refactoring of StoreAttrDefault so that some code could be shared.
>
> I think this is probably OK now, except for one thing: you're likely
> to have issues if a dropped column has an attmissingval, because often
> the value won't agree with the bogus "integer" type we use for those;
> worse, the array might refer to a type that no longer exists.
> One way to improve that is to change
>
> "CASE WHEN a.atthasmissing THEN a.attmissingval "
> "ELSE null END AS attmissingval "
>
> to
>
> "CASE WHEN a.atthasmissing AND NOT a.attisdropped THEN a.attmissingval "
> "ELSE null END AS attmissingval "
>
> However, this might be another reason to reconsider the decision to use
> anyarray: it could easily lead to situations where harmless-looking
> queries like "select * from pg_attribute" fail. Or maybe we should tweak
> DROP COLUMN so that it forcibly resets atthasmissing/attmissingval along
> with setting atthasdropped, so that the case can't arise.
>
> Like Andres, I haven't actually tested, just eyeballed it.
>
>
I moved the bulk of the code into a function in heap.c where it seemed
right at home.
I added removal of missing values from dropped columns, as well as
qualifying the test as above.
The indent issue raised by Alvaro is also fixed. I included your patch
fixing the regression tests.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 44a5c0a917b..9c709315192 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9eea1ed315a..7bffe09fb5d 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bea35afc4bd..69d97bb8ece 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -204,9 +204,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -221,17 +222,18 @@ typedef struct PruneResult
/* Number of newly frozen tuples */
int nfrozen;
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--rdqtp5puvxqotfdw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--tez7m2a73jtztiij
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 6bd8400b33b..abf6bdb2d99 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
}
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..45c4ae22e6a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--racicctn4wry6xe5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 44a5c0a917b..9c709315192 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9eea1ed315a..7bffe09fb5d 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bea35afc4bd..69d97bb8ece 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -204,9 +204,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -221,17 +222,18 @@ typedef struct PruneResult
/* Number of newly frozen tuples */
int nfrozen;
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--rdqtp5puvxqotfdw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--tez7m2a73jtztiij
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 6bd8400b33b..abf6bdb2d99 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
}
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..45c4ae22e6a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--racicctn4wry6xe5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 44a5c0a917b..9c709315192 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9eea1ed315a..7bffe09fb5d 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bea35afc4bd..69d97bb8ece 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -204,9 +204,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -221,17 +222,18 @@ typedef struct PruneResult
/* Number of newly frozen tuples */
int nfrozen;
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--rdqtp5puvxqotfdw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--tez7m2a73jtztiij
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 6bd8400b33b..abf6bdb2d99 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
}
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..45c4ae22e6a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--racicctn4wry6xe5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v9 08/21] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 52 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 224 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 6abfe36dec7..a793c0f56ee 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1106,7 +1106,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index eb09713311b..312695f806c 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -590,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -855,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -898,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -921,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 9d047621ea5..de11c166575 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
-/* 'reason' codes for heap_page_prune() */
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -361,13 +365,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfa9d5aaeac..5737bc5b945 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2191,8 +2191,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--caj67xgx3lukmr5f
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v7 08/16] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 53 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 225 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2b7c7026429..4802d789963 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1048,7 +1048,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 457650ab651..e009c7579dd 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -594,7 +705,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -859,10 +970,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -902,7 +1013,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -925,7 +1036,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 59c81f38e51..6f9c66a872b 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
-/* 'reason' codes for heap_page_prune() */
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
+
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -332,6 +336,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -358,13 +363,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4679660837c..cc6a33ab3ee 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2192,8 +2192,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--ck6erxojvlx23byk
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v9 08/21] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 52 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 224 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 6abfe36dec7..a793c0f56ee 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1106,7 +1106,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index eb09713311b..312695f806c 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -590,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -855,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -898,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -921,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 9d047621ea5..de11c166575 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
-/* 'reason' codes for heap_page_prune() */
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -361,13 +365,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfa9d5aaeac..5737bc5b945 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2191,8 +2191,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--caj67xgx3lukmr5f
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* [PATCH v7 08/16] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 53 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 225 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2b7c7026429..4802d789963 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1048,7 +1048,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 457650ab651..e009c7579dd 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -594,7 +705,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -859,10 +970,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -902,7 +1013,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -925,7 +1036,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 59c81f38e51..6f9c66a872b 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
-/* 'reason' codes for heap_page_prune() */
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
+
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -332,6 +336,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -358,13 +363,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4679660837c..cc6a33ab3ee 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2192,8 +2192,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--ck6erxojvlx23byk
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 57+ messages in thread
* Re: Add LSN <-> time conversion functionality
@ 2024-06-27 01:35 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 57+ messages in thread
From: Melanie Plageman @ 2024-06-27 01:35 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: pgsql-hackers; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>
On Mon, Mar 18, 2024 at 1:29 PM Tomas Vondra
<[email protected]> wrote:
>
> On 2/22/24 03:45, Melanie Plageman wrote:
> > The attached v3 has a new algorithm. Now, LSNTimes are added from the
> > end of the array forward until all array elements have at least one
> > logical member (array length == volume). Once array length == volume,
> > new LSNTimes will result in merging logical members in existing
> > elements. We want to merge older members because those can be less
> > precise. So, the number of logical members per array element will
> > always monotonically increase starting from the beginning of the array
> > (which contains the most recent data) and going to the end. We want to
> > use all the available space in the array. That means that each LSNTime
> > insertion will always result in a single merge. We want the timeline
> > to be inclusive of the oldest data, so merging means taking the
> > smaller value of two LSNTime values. I had to pick a rule for choosing
> > which elements to merge. So, I choose the merge target as the oldest
> > element whose logical membership is < 2x its predecessor. I merge the
> > merge target's predecessor into the merge target. Then I move all of
> > the intervening elements down 1. Then I insert the new LSNTime at
> > index 0.
> >
>
> I can't help but think about t-digest [1], which also merges data into
> variable-sized buckets (called centroids, which is a pair of values,
> just like LSNTime). But the merging is driven by something called "scale
> function" which I found like a pretty nice approach to this, and it
> yields some guarantees regarding accuracy. I wonder if we could do
> something similar here ...
>
> The t-digest is a way to approximate quantiles, and the default scale
> function is optimized for best accuracy on the extremes (close to 0.0
> and 1.0), but it's possible to use scale functions that optimize only
> for accuracy close to 1.0.
>
> This may be misguided, but I see similarity between quantiles and what
> LSNTimeline does - timestamps are ordered, and quantiles close to 0.0
> are "old timestamps" while quantiles close to 1.0 are "now".
>
> And t-digest also defines a pretty efficient algorithm to merge data in
> a way that gradually combines older "buckets" into larger ones.
I started taking a look at this paper and think the t-digest could be
applicable as a possible alternative data structure to the one I am
using to approximate page age for the actual opportunistic freeze
heuristic -- especially since the centroids are pairs of a mean and a
count. I couldn't quite understand how the t-digest is combining those
centroids. Since I am not computing quantiles over the LSNTimeStream,
though, I think I can probably do something simpler for this part of
the project.
> >> - The LSNTimeline comment claims an array of size 64 is large enough to
> >> not need to care about filling it, but maybe it should briefly explain
> >> why we can never fill it (I guess 2^64 is just too many).
-- snip --
> I guess that should be enough for (2^64-1) logical members, because it's
> a sequence 1, 2, 4, 8, ..., 2^63. Seems enough.
>
> But now that I think about it, does it make sense to do the merging
> based on the number of logical members? Shouldn't this really be driven
> by the "length" of the time interval the member covers?
After reading this, I decided to experiment with a few different
algorithms in python and plot the unabbreviated LSNTimeStream against
various ways of compressing it. You can see the results if you run the
python code here [1].
What I found is that attempting to calculate the error represented by
dropping a point and picking the point which would cause the least
additional error were it to be dropped produced more accurate results
than combining the oldest entries based on logical membership to fit
some series.
This is inspired by what you said about using the length of segments
to decide which points to merge. In my case, I want to merge segments
that have a similar slope -- those which have a point that is
essentially redundant. I loop through the LSNTimeStream and look for
the point that we can drop with the lowest impact on future
interpolation accuracy. To do this, I calculate the area of the
triangle made by each point on the stream and its adjacent points. The
idea being that if you drop that point, the triangle is the amount of
error you introduce for points being interpolated between the new pair
(previous adjacencies of the dropped point). This has some issues, but
it seems more logical than just always combining the oldest points.
If you run the python simulator code, you'll see that for the
LSNTimeStream I generated, using this method produces more accurate
results than either randomly dropping points or using the "combine
oldest members" method.
It would be nice if we could do something with the accumulated error
-- so we could use it to modify estimates when interpolating. I don't
really know how to keep it though. I thought I would just save the
calculated error in one or the other of the adjacent points after
dropping a point, but then what do we do with the error saved in a
point before it is dropped? Add it to the error value in one of the
adjacent points? If we did, what would that even mean? How would we
use it?
- Melanie
[1] https://gist.github.com/melanieplageman/95126993bcb43d4b4042099e9d0ccc11
^ permalink raw reply [nested|flat] 57+ messages in thread
end of thread, other threads:[~2024-06-27 01:35 UTC | newest]
Thread overview: 57+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-06-19 14:55 Fast default stuff versus pg_upgrade Tom Lane <[email protected]>
2018-06-19 15:51 ` Andrew Dunstan <[email protected]>
2018-06-19 16:05 ` Andres Freund <[email protected]>
2018-06-19 16:08 ` Tom Lane <[email protected]>
2018-06-19 16:17 ` Andrew Dunstan <[email protected]>
2018-06-19 16:36 ` Andres Freund <[email protected]>
2018-06-19 16:17 ` Tom Lane <[email protected]>
2018-06-19 16:33 ` Andres Freund <[email protected]>
2018-06-19 16:37 ` Tom Lane <[email protected]>
2018-06-19 16:52 ` Andres Freund <[email protected]>
2018-06-19 17:19 ` Tom Lane <[email protected]>
2018-06-19 17:39 ` Andrew Dunstan <[email protected]>
2018-06-20 02:41 ` Andrew Dunstan <[email protected]>
2018-06-20 02:46 ` Andres Freund <[email protected]>
2018-06-20 16:51 ` Andrew Dunstan <[email protected]>
2018-06-20 16:58 ` Andres Freund <[email protected]>
2018-06-20 17:46 ` Andrew Dunstan <[email protected]>
2018-06-21 00:53 ` Andrew Dunstan <[email protected]>
2018-06-21 01:04 ` Andres Freund <[email protected]>
2018-06-21 12:07 ` Andrew Dunstan <[email protected]>
2018-06-21 16:04 ` Tom Lane <[email protected]>
2018-06-21 16:08 ` Tom Lane <[email protected]>
2018-06-21 17:31 ` Andrew Dunstan <[email protected]>
2018-06-21 16:30 ` Andres Freund <[email protected]>
2018-06-21 16:39 ` Tom Lane <[email protected]>
2018-06-21 17:02 ` Andrew Dunstan <[email protected]>
2018-06-21 17:18 ` Tom Lane <[email protected]>
2018-06-21 17:28 ` Andrew Dunstan <[email protected]>
2018-06-21 17:40 ` Andres Freund <[email protected]>
2018-06-21 17:44 ` Tom Lane <[email protected]>
2018-06-21 17:46 ` Andres Freund <[email protected]>
2018-06-21 17:50 ` Tom Lane <[email protected]>
2018-06-21 17:49 ` Andrew Dunstan <[email protected]>
2018-06-21 17:53 ` Tom Lane <[email protected]>
2018-06-21 18:51 ` Andrew Dunstan <[email protected]>
2018-06-21 20:41 ` Tom Lane <[email protected]>
2018-06-22 13:01 ` Andrew Dunstan <[email protected]>
2018-06-21 21:10 ` Alvaro Herrera <[email protected]>
2018-06-21 21:20 ` Tom Lane <[email protected]>
2018-06-19 17:12 ` Robert Haas <[email protected]>
2018-06-19 17:23 ` Peter Geoghegan <[email protected]>
2018-06-19 17:41 ` David G. Johnston <[email protected]>
2018-06-19 17:59 ` Tom Lane <[email protected]>
2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v7 08/16] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v7 08/16] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v9 08/21] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v9 08/21] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-06-27 01:35 Re: Add LSN <-> time conversion functionality Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox