pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedrelfrozenxid may disagree with row XIDs after 1ccc1e05ae
62+ messages / 8 participants
[nested] [flat]
Commits from this thread:
2024-07-19 e2e992820f d42f60ccf0 01e2b7f0fd Test that vacuum removes tuples older than OldestXmin
2024-07-19 dc6354c670 45ce054c02 06bf404cd0 Ensure vacuum removes all visibly dead tuples older than OldestXmin
* relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-04 00:07 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-03-04 00:07 UTC (permalink / raw)
To: robertmhaas@gmail.com; melanieplageman@gmail.com; +Cc: Peter Geoghegan <pg@bowt.ie>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Sat, Jan 06, 2024 at 02:44:48PM -0800, Noah Misch wrote:
> On Sat, Jan 06, 2024 at 01:30:40PM -0800, Peter Geoghegan wrote:
> > On Sat, Jan 6, 2024 at 12:24 PM Noah Misch <noah@leadboat.com> wrote:
> > > Fair enough. While I agree there's a decent chance back-patching would be
> > > okay, I think there's also a decent chance that 1ccc1e05ae creates the problem
> > > Matthias theorized. Something like: we update relfrozenxid based on
> > > OldestXmin, even though GlobalVisState caused us to retain a tuple older than
> > > OldestXmin. Then relfrozenxid disagrees with table contents.
I figure Matthias's upthread theory is more likely than not to hold. If it
does hold, commit 1ccc1e05ae created a new corruption route. Hence, I'm
adding a v17 open item for commit 1ccc1e05ae.
> > Either every relevant code path has the same OldestXmin to work off
> > of, or the whole NewRelfrozenXid/relfrozenxid-tracking thing can't be
> > expected to work as designed. I find it a bit odd that
> > pruneheap.c/GlobalVisState has no direct understanding of this
> > dependency (none that I can discern, at least). Wouldn't it at least
> > be more natural if pruneheap.c could access OldestXmin when run inside
> > VACUUM? (Could just be used by defensive hardening code.)
>
> Tied to that decision is the choice of semantics when the xmin horizon moves
> backward during one VACUUM, e.g. when a new walsender xmin does so. Options:
>
> 1. Continue to remove tuples based on the OldestXmin from VACUUM's start. We
> could have already removed some of those tuples, so the walsender xmin
> won't achieve a guarantee anyway. (VACUUM would want ratchet-like behavior
> in GlobalVisState, possibly by sharing OldestXmin with pruneheap like you
> say.)
>
> 2. Move OldestXmin backward, to reflect the latest xmin horizon. (Perhaps
> VACUUM would just pass GlobalVisState to a function that returns the
> compatible OldestXmin.)
>
> Which way is better?
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-21 16:15 Robert Haas <robertmhaas@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 62+ messages in thread
From: Robert Haas @ 2024-03-21 16:15 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: melanieplageman@gmail.com, Peter Geoghegan <pg@bowt.ie>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Sun, Mar 3, 2024 at 7:07 PM Noah Misch <noah@leadboat.com> wrote:
> I figure Matthias's upthread theory is more likely than not to hold. If it
> does hold, commit 1ccc1e05ae created a new corruption route. Hence, I'm
> adding a v17 open item for commit 1ccc1e05ae.
I need some help understanding what's going on here. I became aware of
this thread because I took a look at the open items list.
This email seems to have branched off of the thread for bug #17257,
reported 2021-10-29. The antecedent of "Matthias's upthread theory" is
unclear to me. These emails seem like the most relevant ones:
https://www.postgresql.org/message-id/CAEze2Wj7O5tnM_U151Baxr5ObTJafwH%3D71_JEmgJV%2B6eBgjL7g%40mail...
https://www.postgresql.org/message-id/CAEze2WhxhEQEx%2Bc%2BCXoDpQs1H1HgkYUK4BW-hFw5_eQxuVWqRw%40mail...
https://www.postgresql.org/message-id/20240106202413.e5%40rfd.leadboat.com
But I'm having a hard time piecing it all together. The general
picture seems to be that pruning and vacuum disagree about whether a
particular tuple is prunable; before 1ccc1e05ae, that caused the retry
loop in heap_page_prune() to retry forever. Now, it causes
relfrozenxid to be set to too new a value, which is a data-corruption
scenario. If that's right, I'm slightly miffed to find this being
labeled as an open item, since that makes it seem like 1ccc1e05ae
didn't create any new problem but only caused existing defects in the
GlobalVisTest machinery to have different consequences. Perhaps it's
all for the best, though. It's kind of embarrassing that we haven't
fixed whatever the problem is here yet.
But what exactly is the problem, and what's the fix? In the first of
the emails linked above, Matthias argues that the problem is that
GlobalVisState->maybe_needed can move backward. Peter Geoghegan seems
to agree with that here:
https://www.postgresql.org/message-id/CAH2-Wzk_L7Z7LREHTtg5vY08eeWdnHO70m98eWx4U1uwvW%3D0sA%40mail.g...
And Peter seems to have been trying to make sense of Andres's remarks
here, which I think are saying the same thing:
https://www.postgresql.org/message-id/20210616192202.6q63mu66h4uyn343%40alap3.anarazel.de
So it seems like Matthias, Peter, and Andres all agree that
GlobalVisState->maybe_needed going backward is bad and causes this
problem. Unfortunately, I don't understand the mechanism.
vacuumlazy.c's LVRelState has this:
/* VACUUM operation's cutoffs for freezing and pruning */
struct VacuumCutoffs cutoffs;
GlobalVisState *vistest;
/* Tracks oldest extant XID/MXID for setting relfrozenxid/relminmxid */
TransactionId NewRelfrozenXid;
MultiXactId NewRelminMxid;
This looks scary, because we've got an awful lot of different
variables serving rather closely-related purposes. cutoffs actually
contains three separate XID/MXID pairs; GlobalVisTest contains XIDs;
and then we've got another XID/MXID pair in the form of
NewRel{frozenXid,minMxid}. It's certainly understandable how there
could be a bug here: you just need whichever of these things
ultimately goes into relfrozenxid to disagree with whichever of these
things actually controls the pruning behavior. But it is not clear to
me exactly which of these things disagree with each other in exactly
what way, and, perhaps as a result, it is also not clear to my why
preventing GlobalVisTest->maybe_needed from moving backward would fix
anything.
Can someone explain, or point me to the relevant previous email?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-21 17:21 Matthias van de Meent <boekewurm+postgres@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Matthias van de Meent @ 2024-03-21 17:21 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; melanieplageman@gmail.com, Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Thu, 21 Mar 2024 at 17:15, Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Sun, Mar 3, 2024 at 7:07 PM Noah Misch <noah@leadboat.com> wrote:
> > I figure Matthias's upthread theory is more likely than not to hold. If it
> > does hold, commit 1ccc1e05ae created a new corruption route. Hence, I'm
> > adding a v17 open item for commit 1ccc1e05ae.
>
> I need some help understanding what's going on here. I became aware of
> this thread because I took a look at the open items list.
>
> This email seems to have branched off of the thread for bug #17257,
> reported 2021-10-29. The antecedent of "Matthias's upthread theory" is
> unclear to me. These emails seem like the most relevant ones:
>
> https://www.postgresql.org/message-id/CAEze2Wj7O5tnM_U151Baxr5ObTJafwH%3D71_JEmgJV%2B6eBgjL7g%40mail...
> https://www.postgresql.org/message-id/CAEze2WhxhEQEx%2Bc%2BCXoDpQs1H1HgkYUK4BW-hFw5_eQxuVWqRw%40mail...
> https://www.postgresql.org/message-id/20240106202413.e5%40rfd.leadboat.com
>
> But I'm having a hard time piecing it all together. The general
> picture seems to be that pruning and vacuum disagree about whether a
> particular tuple is prunable; before 1ccc1e05ae, that caused the retry
> loop in heap_page_prune() to retry forever. Now, it causes
> relfrozenxid to be set to too new a value, which is a data-corruption
> scenario. If that's right, I'm slightly miffed to find this being
> labeled as an open item, since that makes it seem like 1ccc1e05ae
> didn't create any new problem but only caused existing defects in the
> GlobalVisTest machinery to have different consequences. Perhaps it's
> all for the best, though. It's kind of embarrassing that we haven't
> fixed whatever the problem is here yet.
>
> But what exactly is the problem, and what's the fix? In the first of
> the emails linked above, Matthias argues that the problem is that
> GlobalVisState->maybe_needed can move backward. Peter Geoghegan seems
> to agree with that here:
>
> https://www.postgresql.org/message-id/CAH2-Wzk_L7Z7LREHTtg5vY08eeWdnHO70m98eWx4U1uwvW%3D0sA%40mail.g...
>
> And Peter seems to have been trying to make sense of Andres's remarks
> here, which I think are saying the same thing:
>
> https://www.postgresql.org/message-id/20210616192202.6q63mu66h4uyn343%40alap3.anarazel.de
>
> So it seems like Matthias, Peter, and Andres all agree that
> GlobalVisState->maybe_needed going backward is bad and causes this
> problem. Unfortunately, I don't understand the mechanism.
There are 2 mechanisms I know of which allow this value to go backwards:
1. Replication slots that connect may set their backend's xmin to an
xmin < GlobalXmin.
This is known and has been documented, and was considered OK when this
was discussed on the list previously.
2. The commit abort path has a short window in which the backend's
xmin is unset and does not mirror the xmin of registered snapshots.
This is what I described in [0], and may be the worst (?) offender.
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
[0] https://www.postgresql.org/message-id/CAEze2Wj%2BV0kTx86xB_YbyaqTr5hnE_igdWAwuhSyjXBYscf5-Q%40mail.g...
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-22 12:22 Robert Haas <robertmhaas@gmail.com>
parent: Matthias van de Meent <boekewurm+postgres@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Robert Haas @ 2024-03-22 12:22 UTC (permalink / raw)
To: Matthias van de Meent <boekewurm+postgres@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; melanieplageman@gmail.com, Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Thu, Mar 21, 2024 at 1:22 PM Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
> > So it seems like Matthias, Peter, and Andres all agree that
> > GlobalVisState->maybe_needed going backward is bad and causes this
> > problem. Unfortunately, I don't understand the mechanism.
>
> There are 2 mechanisms I know of which allow this value to go backwards:
I actually wasn't asking about the mechanism by which
GlobalVisState->maybe_needed could go backwards. I was asking about
the mechanism by which that could cause bad things to happen.
> 1. Replication slots that connect may set their backend's xmin to an
> xmin < GlobalXmin.
> This is known and has been documented, and was considered OK when this
> was discussed on the list previously.
Right, OK.
> 2. The commit abort path has a short window in which the backend's
> xmin is unset and does not mirror the xmin of registered snapshots.
> This is what I described in [0], and may be the worst (?) offender.
>
> [0] https://www.postgresql.org/message-id/CAEze2Wj%2BV0kTx86xB_YbyaqTr5hnE_igdWAwuhSyjXBYscf5-Q%40mail.g...
So, what I would say is that this sounds inadvertent and so perhaps we
should do something about it, but also, it seems wrong to me that it
causes any serious problem. As far as I know, we've always treated the
result of an xmin calculation going backward as a rare but expected
case with which everything that depends on xmin calculations must
cope.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-22 18:41 Melanie Plageman <melanieplageman@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-03-22 18:41 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 22, 2024 at 8:22 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Mar 21, 2024 at 1:22 PM Matthias van de Meent
> <boekewurm+postgres@gmail.com> wrote:
> > > So it seems like Matthias, Peter, and Andres all agree that
> > > GlobalVisState->maybe_needed going backward is bad and causes this
> > > problem. Unfortunately, I don't understand the mechanism.
> >
> > There are 2 mechanisms I know of which allow this value to go backwards:
>
> I actually wasn't asking about the mechanism by which
> GlobalVisState->maybe_needed could go backwards. I was asking about
> the mechanism by which that could cause bad things to happen.
>
> > 1. Replication slots that connect may set their backend's xmin to an
> > xmin < GlobalXmin.
> > This is known and has been documented, and was considered OK when this
> > was discussed on the list previously.
>
> Right, OK.
>
> > 2. The commit abort path has a short window in which the backend's
> > xmin is unset and does not mirror the xmin of registered snapshots.
> > This is what I described in [0], and may be the worst (?) offender.
> >
> > [0] https://www.postgresql.org/message-id/CAEze2Wj%2BV0kTx86xB_YbyaqTr5hnE_igdWAwuhSyjXBYscf5-Q%40mail.g...
>
> So, what I would say is that this sounds inadvertent and so perhaps we
> should do something about it, but also, it seems wrong to me that it
> causes any serious problem. As far as I know, we've always treated the
> result of an xmin calculation going backward as a rare but expected
> case with which everything that depends on xmin calculations must
> cope.
I'm still catching up here, so forgive me if this is a dumb question:
Does using GlobalVisState instead of VacuumCutoffs->OldestXmin when
freezing and determining relfrozenxid not solve the problem?
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-22 19:34 Robert Haas <robertmhaas@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 1 reply; 62+ messages in thread
From: Robert Haas @ 2024-03-22 19:34 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 22, 2024 at 2:41 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
> I'm still catching up here, so forgive me if this is a dumb question:
> Does using GlobalVisState instead of VacuumCutoffs->OldestXmin when
> freezing and determining relfrozenxid not solve the problem?
I think we need to understand what's actually happening here before we
jump to solutions. I think it's clear that we can't determine a
relfrozenxid in a way that is untethered from the decisions made while
pruning, but surely whoever wrote this code intended for there to be
some relationship between GlobalVisState and
VacuumCutoffs->OldestXmin. For example, it may be that they intended
for VacuumCutoffs->OldestXmin to always contain an XID that is older
than any XID that the GlobalVisState could possibly have pruned; but
they might have done that incorrectly, in such a fashion that when
GlobalVisState->maybe_needed moves backward the intended invariant no
longer holds.
But I think that we surely cannot think that they just threw some XID
that fell off the cart into VacuumCutoffs->OldestXmin and some other
XID that they pulled out of the air into GlobalVisState and hoped
everything worked out. It seems really likely that they had some
reason for thinking that we needed two things here rather than just
one. If not, OK, let's get rid of one of them and make it simpler, but
if we just assume that there's no real reason for having both fields
here and simplify accordingly, we're probably just going to be trading
this bug for some other one.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-22 19:43 Noah Misch <noah@leadboat.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 0 replies; 62+ messages in thread
From: Noah Misch @ 2024-03-22 19:43 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 22, 2024 at 02:41:25PM -0400, Melanie Plageman wrote:
> On Fri, Mar 22, 2024 at 8:22 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > On Thu, Mar 21, 2024 at 1:22 PM Matthias van de Meent
> > <boekewurm+postgres@gmail.com> wrote:
> > > > So it seems like Matthias, Peter, and Andres all agree that
> > > > GlobalVisState->maybe_needed going backward is bad and causes this
> > > > problem. Unfortunately, I don't understand the mechanism.
> > >
> > > There are 2 mechanisms I know of which allow this value to go backwards:
> >
> > I actually wasn't asking about the mechanism by which
> > GlobalVisState->maybe_needed could go backwards. I was asking about
> > the mechanism by which that could cause bad things to happen.
> >
> > > 1. Replication slots that connect may set their backend's xmin to an
> > > xmin < GlobalXmin.
> > > This is known and has been documented, and was considered OK when this
> > > was discussed on the list previously.
> >
> > Right, OK.
> >
> > > 2. The commit abort path has a short window in which the backend's
> > > xmin is unset and does not mirror the xmin of registered snapshots.
> > > This is what I described in [0], and may be the worst (?) offender.
> > >
> > > [0] https://www.postgresql.org/message-id/CAEze2Wj%2BV0kTx86xB_YbyaqTr5hnE_igdWAwuhSyjXBYscf5-Q%40mail.g...
> >
> > So, what I would say is that this sounds inadvertent and so perhaps we
> > should do something about it, but also, it seems wrong to me that it
> > causes any serious problem. As far as I know, we've always treated the
> > result of an xmin calculation going backward as a rare but expected
> > case with which everything that depends on xmin calculations must
> > cope.
>
> I'm still catching up here, so forgive me if this is a dumb question:
> Does using GlobalVisState instead of VacuumCutoffs->OldestXmin when
> freezing and determining relfrozenxid not solve the problem?
One could fix it along those lines. If GlobalVisState moves forward during
VACUUM, that's fine, but relfrozenxid needs to reflect the overall outcome,
not just the final GlobalVisState. Suppose we remove XIDs <100 at page 1, <99
at page 2, and <101 at page 3. relfrozenxid needs the value it would get if
we had removed <99 at every page. I think GlobalVisState doesn't track that
today, but it could. The 2024-03-14 commit e85662d added
GetStrictOldestNonRemovableTransactionId(), which targets a similar problem.
I've not reviewed it, but I suggest checking it for relevance to $SUBJECT.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-29 16:05 Robert Haas <robertmhaas@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 3 replies; 62+ messages in thread
From: Robert Haas @ 2024-03-29 16:05 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 22, 2024 at 3:34 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I think we need to understand what's actually happening here before we
> jump to solutions. I think it's clear that we can't determine a
> relfrozenxid in a way that is untethered from the decisions made while
> pruning, but surely whoever wrote this code intended for there to be
> some relationship between GlobalVisState and
> VacuumCutoffs->OldestXmin. For example, it may be that they intended
> for VacuumCutoffs->OldestXmin to always contain an XID that is older
> than any XID that the GlobalVisState could possibly have pruned; but
> they might have done that incorrectly, in such a fashion that when
> GlobalVisState->maybe_needed moves backward the intended invariant no
> longer holds.
OK, I did some more analysis of this. I kind of feel like maybe this
should be getting more attention from Melanie (since it was the commit
of her patch that triggered the addition of this to the open items
list) or Andres (because it has been alleged to be related to the
GlobalVisTest stuff), but here we are.
Maybe this is already clear to some other people here, but I think the
bug is essentially the result of this code in heap_vacuum_rel:
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
vacrel->vistest = GlobalVisTestFor(rel);
The idea here seems to be that because we call vacuum_get_cutoffs()
before we call GlobalVisTestFor(), the cutoff established by the
former should precede the cutoff established by the latter.
vacrel->cutoffs.OldestXmin is propagated into vacrel->NewRelfrozenXid,
and although the latter value seems like it can get updated later, it
remains true that vacrel->cutoffs.OldestXmin affects how relfrozenxid
ultimately gets set. However, that value is set in
vacuum_get_cutoffs() to the result of
GetOldestNonRemovableTransactionId(rel); and that function works with
the results of ComputeXidHorizons(). Meanwhile, GlobalVisTestFor()
creates a visibility test object which is ultimately going to be
tested inside heap_page_prune() using GlobalVisTestIsRemovableXid()
which is a wrapper around GlobalVisTestIsRemovableFullXid() which can
call GlobalVisUpdate() which can call ... ta da ...
ComputeXidHorizons(). So the behavior of ComputeXidHorizons() is the
key point here. And sure enough, the header comment for that function
says this:
* Note: despite the above, it's possible for the calculated values to move
* backwards on repeated calls. The calculated values are conservative, so
* that anything older is definitely not considered as running by anyone
* anymore, but the exact values calculated depend on a number of things. For
* example, if there are no transactions running in the current database, the
* horizon for normal tables will be latestCompletedXid. If a transaction
* begins after that, its xmin will include in-progress transactions in other
* databases that started earlier, so another call will return a lower value.
* Nonetheless it is safe to vacuum a table in the current database with the
* first result. There are also replication-related effects: a walsender
* process can set its xmin based on transactions that are no longer running
* on the primary but are still being replayed on the standby, thus possibly
* making the values go backwards. In this case there is a possibility that
* we lose data that the standby would like to have, but unless the standby
* uses a replication slot to make its xmin persistent there is little we can
* do about that --- data is only protected if the walsender runs continuously
* while queries are executed on the standby. (The Hot Standby code deals
* with such cases by failing standby queries that needed to access
* already-removed data, so there's no integrity bug.)
That means that heap_vacuum_rel() isn't guaranteeing anything by
calling vacuum_get_cutoffs() before it calls GlobalVisTestFor(). I
started wondering when this bug got introduced, and concluded that it
probably dates all the way back to
dc7420c2c9274a283779ec19718d2d16323640c0, when Andres first introduced
the GlobalVisTest stuff. The code was substantially refactored
afterwards by Peter Geoghegan, in
b4af70cb210393c9c8f41643acf6b213e21178e7 and
73f6ec3d3c8d5786c54373e71a096e5acf78e7ca. However, as far as I can
see, neither of those commits created the problem; they just moved
code around, and in fact I'd say that the problem is easier to see now
than it was before all of that refactoring because the two problematic
initializations are now closer together than they were originally. I
think that Peter may have been under the impression at the time of the
latter commit that the order of initializing vacrel->cutoffs vs.
vacrel->vistest was significant, because the commit message says:
Explain the relationship between vacuumlazy.c's vistest and OldestXmin
cutoffs. These closely related cutoffs are different in subtle but
important ways. Also document a closely related rule: we must establish
rel_pages _after_ OldestXmin to ensure that no XID < OldestXmin can be
missed by lazy_scan_heap().
And a comment in that commit says:
+ * We expect vistest will always make heap_page_prune remove any deleted
+ * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
Neither of these is perhaps 100% explicit about the ordering being
relevant, but if that wasn't thought to make this code safe, then I
don't know what was thought to make it safe.
At any rate, unless I'm missing something, which is definitely
possible, the actual bug dates to dc7420c2c, which means whatever fix
we come up with here will need to be back-patched back to v14.
But what exactly is the fix here? I now understand why there was
discussion of preventing maybe_needed from going backward, but I'm not
sure that's quite the right idea. I think the problem isn't exactly
that the maybe_needed value that is updated by GlobalVisUpdateApply()
can go backwards, but rather that the SOMETHING_oldest_removable value
on which it depends can go backwards. AFAICS, the problem isn't so
much that the GlobalVisTest itself can retreat to an older value, but
that it can retreat compared to the OldestXmin obtained from calling
GetOldestNonRemovableTransactionId(), which depends on the
SOMETHING_oldest_removable threshold but *not* on maybe_needed.
Perhaps, for some reason I'm not grokking at the moment, preventing
maybe_needed from retreating would be good enough to prevent trouble
in practice, but it doesn't appear to me to be the most principled
solution as of this writing.
I kind of feel like the code is just going at this in a way that's
completely backwards. It feels like what we should be doing is looking
at the actual states of the pages post-pruning, and specifically what
the oldest (M)XIDs that are really present there are. But changing
that seems like a lot to do in a fix that needs to be back-patched.
It's tempting to look for a narrower fix, where we just find out the
oldest horizon that was ever actually used. But I don't feel very good
about doing that by tampering with ComputeXidHorizons(), either. That
seems like it could have nasty side effects in terms of either
performance or correctness, which is really scary for a fix that needs
to be back-patched. I wonder if there's some way that we can get a
more reliable idea of what XIDs were used for pruning that is strictly
local to vacuum and can't affect the rest of the system (and is also
hopefully not so conservative that it causes damage by unduly reducing
relfrozenxid advancement). However, I don't have a good idea about
what such a fix might look like right now, and I'm out of time for
today.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-29 17:16 Peter Geoghegan <pg@bowt.ie>
parent: Robert Haas <robertmhaas@gmail.com>
2 siblings, 1 reply; 62+ messages in thread
From: Peter Geoghegan @ 2024-03-29 17:16 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 29, 2024 at 12:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> That means that heap_vacuum_rel() isn't guaranteeing anything by
> calling vacuum_get_cutoffs() before it calls GlobalVisTestFor(). I
> started wondering when this bug got introduced, and concluded that it
> probably dates all the way back to
> dc7420c2c9274a283779ec19718d2d16323640c0, when Andres first introduced
> the GlobalVisTest stuff. The code was substantially refactored
> afterwards by Peter Geoghegan, in
> b4af70cb210393c9c8f41643acf6b213e21178e7 and
> 73f6ec3d3c8d5786c54373e71a096e5acf78e7ca. However, as far as I can
> see, neither of those commits created the problem; they just moved
> code around, and in fact I'd say that the problem is easier to see now
> than it was before all of that refactoring because the two problematic
> initializations are now closer together than they were originally. I
> think that Peter may have been under the impression at the time of the
> latter commit that the order of initializing vacrel->cutoffs vs.
> vacrel->vistest was significant, because the commit message says:
FWIW I never thought that the order that we called
vacuum_get_cutoffs() relative to when we call GlobalVisTestFor() was
directly significant (though I did think that about the order that we
attain VACUUM's rel_pages and the vacuum_get_cutoffs() call). I can't
have thought that, because clearly GlobalVisTestFor() just returns a
pointer, and so cannot directly affect backend local state.
> Explain the relationship between vacuumlazy.c's vistest and OldestXmin
> cutoffs. These closely related cutoffs are different in subtle but
> important ways. Also document a closely related rule: we must establish
> rel_pages _after_ OldestXmin to ensure that no XID < OldestXmin can be
> missed by lazy_scan_heap().
>
> And a comment in that commit says:
>
> + * We expect vistest will always make heap_page_prune remove any deleted
> + * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
>
> Neither of these is perhaps 100% explicit about the ordering being
> relevant, but if that wasn't thought to make this code safe, then I
> don't know what was thought to make it safe.
I intended to highlight the general nature of the dependency --
nothing more. I don't claim to be an authoritative source on this,
though I was confident that the code was *supposed* to work like this
(how else could it possibly work?).
It was clear that this is an important issue, from an early stage.
Pre-release 14 had 2 or 3 bugs that all had the same symptom:
lazy_scan_prune would loop forever. This was true even though each of
the bugs had fairly different underlying causes (all tied to
dc7420c2c). I figured that there might well be more bugs like that in
the future.
> At any rate, unless I'm missing something, which is definitely
> possible, the actual bug dates to dc7420c2c, which means whatever fix
> we come up with here will need to be back-patched back to v14.
I agree.
> I kind of feel like the code is just going at this in a way that's
> completely backwards. It feels like what we should be doing is looking
> at the actual states of the pages post-pruning, and specifically what
> the oldest (M)XIDs that are really present there are. But changing
> that seems like a lot to do in a fix that needs to be back-patched.
I said the same thing recently, to Noah. This is still something that
ought to be made very explicit -- pruneheap.c should be directly aware
of its responsibility to not leave any XID < VACUUM's OldestXmin
behind, except for any XIDs that only freezing can deal with. (We're
moving in the direction of making pruneheap.c responsible for
freezing, so perhaps my caveat about freezing shouldn't apply on
HEAD.)
> It's tempting to look for a narrower fix, where we just find out the
> oldest horizon that was ever actually used. But I don't feel very good
> about doing that by tampering with ComputeXidHorizons(), either.
I have every reason to believe that the remaining problems in this
area are extremely rare. I wonder if it would make sense to focus on
making the infinite loop behavior in lazy_scan_prune just throw an
error.
I now fear that that'll be harder than one might think. At the time
that I added the looping behavior (in commit 8523492d), I believed
that the only "legitimate" reason that it could ever be needed was the
same reason why we needed the old tupgone behavior (to deal with
concurrently-inserted tuples from transactions that abort in flight).
But now I worry that it's actually protective, in some way that isn't
generally understood. And so it might be that converting the retry
into a hard error (e.g., erroring-out after MaxHeapTuplesPerPage
retries) will create new problems.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-29 18:27 Robert Haas <robertmhaas@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 62+ messages in thread
From: Robert Haas @ 2024-03-29 18:27 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 29, 2024 at 1:17 PM Peter Geoghegan <pg@bowt.ie> wrote:
> FWIW I never thought that the order that we called
> vacuum_get_cutoffs() relative to when we call GlobalVisTestFor() was
> directly significant (though I did think that about the order that we
> attain VACUUM's rel_pages and the vacuum_get_cutoffs() call). I can't
> have thought that, because clearly GlobalVisTestFor() just returns a
> pointer, and so cannot directly affect backend local state.
Hmm, OK.
> It was clear that this is an important issue, from an early stage.
> Pre-release 14 had 2 or 3 bugs that all had the same symptom:
> lazy_scan_prune would loop forever. This was true even though each of
> the bugs had fairly different underlying causes (all tied to
> dc7420c2c). I figured that there might well be more bugs like that in
> the future.
Looks like you were right.
> I have every reason to believe that the remaining problems in this
> area are extremely rare. I wonder if it would make sense to focus on
> making the infinite loop behavior in lazy_scan_prune just throw an
> error.
>
> I now fear that that'll be harder than one might think. At the time
> that I added the looping behavior (in commit 8523492d), I believed
> that the only "legitimate" reason that it could ever be needed was the
> same reason why we needed the old tupgone behavior (to deal with
> concurrently-inserted tuples from transactions that abort in flight).
> But now I worry that it's actually protective, in some way that isn't
> generally understood. And so it might be that converting the retry
> into a hard error (e.g., erroring-out after MaxHeapTuplesPerPage
> retries) will create new problems.
It also sounds like it would boil down to "ERROR: our code sucks", so
count me as not a fan of that approach. As much as I don't like the
idea of significant changes to the back-branches, I think I like that
idea even less.
On the other hand, I also don't have an idea that I do like right now,
so it's probably too early to decide anything just yet. I'll try to
find more time to study this (and I hope others do the same).
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-30 16:19 Melanie Plageman <melanieplageman@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
2 siblings, 2 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-03-30 16:19 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Fri, Mar 29, 2024 at 12:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 3:34 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > I think we need to understand what's actually happening here before we
> > jump to solutions. I think it's clear that we can't determine a
> > relfrozenxid in a way that is untethered from the decisions made while
> > pruning, but surely whoever wrote this code intended for there to be
> > some relationship between GlobalVisState and
> > VacuumCutoffs->OldestXmin. For example, it may be that they intended
> > for VacuumCutoffs->OldestXmin to always contain an XID that is older
> > than any XID that the GlobalVisState could possibly have pruned; but
> > they might have done that incorrectly, in such a fashion that when
> > GlobalVisState->maybe_needed moves backward the intended invariant no
> > longer holds.
>
> OK, I did some more analysis of this. I kind of feel like maybe this
> should be getting more attention from Melanie (since it was the commit
> of her patch that triggered the addition of this to the open items
> list) or Andres (because it has been alleged to be related to the
> GlobalVisTest stuff), but here we are.
Agreed that I should be doing more of the analysis. I plan to spend
time on this very soon.
Obviously we should actually fix this on back branches, but could we
at least make the retry loop interruptible in some way so people could
use pg_cancel/terminate_backend() on a stuck autovacuum worker or
vacuum process?
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-03-30 17:35 Robert Haas <robertmhaas@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 1 reply; 62+ messages in thread
From: Robert Haas @ 2024-03-30 17:35 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Sat, Mar 30, 2024 at 12:19 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
> Obviously we should actually fix this on back branches, but could we
> at least make the retry loop interruptible in some way so people could
> use pg_cancel/terminate_backend() on a stuck autovacuum worker or
> vacuum process?
Yeah, as long as that's not the only thing we do.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-15 16:35 Robert Haas <robertmhaas@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 3 replies; 62+ messages in thread
From: Robert Haas @ 2024-04-15 16:35 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
Hi,
I propose to remove this open item from
https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
On the original thread (BUG #17257), Alexander Lakhin says that he
can't reproduce this after dad1539ae/18b87b201. Based on my analysis
of the code, I suspect that there is a residual bug, or at least that
there was one prior to 6f47f6883151366c031cd6fd4011e66d2c702a90. (On
the other thread, I cited 6dbb490261a6170a3fc3e326c6983ad63e795047,
but that's not really what I meant.)
But this code is too hairy for me to be certain whether there's a bug
or not without some kind of a test case, and six weeks after the open
item was added, we still don't have one that works on any v16 commit,
either before or after the one named in the subject line. I hate to
leave a potential data-corrupting bug unaddressed, but under present
circumstances, I find it hard to justify spending more time on this.
...Robert
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-15 18:04 Andres Freund <andres@anarazel.de>
parent: Robert Haas <robertmhaas@gmail.com>
2 siblings, 0 replies; 62+ messages in thread
From: Andres Freund @ 2024-04-15 18:04 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-04-15 12:35:59 -0400, Robert Haas wrote:
> I propose to remove this open item from
> https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
>
> On the original thread (BUG #17257), Alexander Lakhin says that he
> can't reproduce this after dad1539ae/18b87b201. Based on my analysis
> of the code, I suspect that there is a residual bug, or at least that
> there was one prior to 6f47f6883151366c031cd6fd4011e66d2c702a90. (On
> the other thread, I cited 6dbb490261a6170a3fc3e326c6983ad63e795047,
> but that's not really what I meant.)
I assume the bug you suspect is that the horizon could go backwards during
pruning, which'd lead to a spurious "found xmin %u from before relfrozenxid
%u" error?
I think Matthias' observation of transaction aborts leading to the xmin
horizon going backward during transaction aborts needs to be fixed - it seems
quite problematic, regardless of whether it causes issues during VACUUM. Even
if it were perfectly safe today, it seems very likely to cause issues in the
future.
> But this code is too hairy for me to be certain whether there's a bug
> or not without some kind of a test case, and six weeks after the open
> item was added, we still don't have one that works on any v16 commit,
> either before or after the one named in the subject line.
Nor does any of the test cases fail on any of the other branches, right?
I.e. they're just testing a bug that's been fixed for ~3 years?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-15 18:52 Andres Freund <andres@anarazel.de>
parent: Robert Haas <robertmhaas@gmail.com>
2 siblings, 1 reply; 62+ messages in thread
From: Andres Freund @ 2024-04-15 18:52 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-03-29 12:05:31 -0400, Robert Haas wrote:
> But what exactly is the fix here? I now understand why there was
> discussion of preventing maybe_needed from going backward, but I'm not
> sure that's quite the right idea. I think the problem isn't exactly
> that the maybe_needed value that is updated by GlobalVisUpdateApply()
> can go backwards, but rather that the SOMETHING_oldest_removable value
> on which it depends can go backwards. AFAICS, the problem isn't so
> much that the GlobalVisTest itself can retreat to an older value, but
> that it can retreat compared to the OldestXmin obtained from calling
> GetOldestNonRemovableTransactionId(), which depends on the
> SOMETHING_oldest_removable threshold but *not* on maybe_needed.
I agree that in theory SOMETHING_oldest_removable should never go
backwards. But in practice it's decidedly nontrivial to prevent that. There's
a reason we've had a version of this comment:
* Note: despite the above, it's possible for the calculated values to move
* backwards on repeated calls.
for a long time.
We really ought to ensure that it doesn't go backwards even across sessions,
because one session vacuuming a relation with one horizon, and then a
subsequent session vacuuming the same relation with an earlier horizon seems
like a recipe for bugs.
But because the horizon depends on the database of the affected object, we
can't just stash a value for each of the possible horizons in shared
memory. We'd need a database-oid keyed hash-table or such.
OTOH, for hot_standby_feedback without a slot, we kinda *do* want the horizons
to go backwards, so that a brief failure of the replication connection is less
likely to cause ongoing queries to fail over the next while.
> Perhaps, for some reason I'm not grokking at the moment, preventing
> maybe_needed from retreating would be good enough to prevent trouble
> in practice, but it doesn't appear to me to be the most principled
> solution as of this writing.
If we prevent maybe_needed from retreating below the OldestXmin value used for
cutoff, then pruning should never be less aggressive than what's needed by
lazy_scan_prune(). So lazy_scan_prune() shouldn't be able to encounter tuples
that weren't considered removable in heap_page_prune(), in < 17, nor would
heap_page_prune_and_freeze() have that issue.
I think one fairly easy fix for this would be to pass OldestXmin to
heap_page_prune[_and_freeze](), and have heap_prune_satisfies_vacuum() first
check OldestXmin and only if that considers the tuple still running, use
GlobalVisTestIsRemovableXid(). That continues to allow us to prune more
aggressively, without any potential risk of IsRemoval being too conservative.
> I kind of feel like the code is just going at this in a way that's
> completely backwards. It feels like what we should be doing is looking
> at the actual states of the pages post-pruning, and specifically what
> the oldest (M)XIDs that are really present there are.
Isn't that pretty much what the code is doing these days? We do additionally
enforce a specific boundary in some cases, to be able to advance relfrozenxid
in some cases, but we advance it more aggressively if there aren't any xids
preventing that (if we scan all the necessary pages etc).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-15 20:52 Noah Misch <noah@leadboat.com>
parent: Robert Haas <robertmhaas@gmail.com>
2 siblings, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-04-15 20:52 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Mon, Apr 15, 2024 at 12:35:59PM -0400, Robert Haas wrote:
> I propose to remove this open item from
> https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
>
> On the original thread (BUG #17257), Alexander Lakhin says that he
> can't reproduce this after dad1539ae/18b87b201. Based on my analysis
I have observed the infinite loop in production with v15.5, so that
non-reproduce outcome is a limitation in the test procedure. (v14.2 added
those two commits.)
> of the code, I suspect that there is a residual bug, or at least that
> there was one prior to 6f47f6883151366c031cd6fd4011e66d2c702a90.
Can you say more about how 6f47f6883151366c031cd6fd4011e66d2c702a90 mitigated
the regression that 1ccc1e05ae introduced? Thanks for discovering that.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-15 21:10 Andres Freund <andres@anarazel.de>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 62+ messages in thread
From: Andres Freund @ 2024-04-15 21:10 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-04-15 13:52:04 -0700, Noah Misch wrote:
> On Mon, Apr 15, 2024 at 12:35:59PM -0400, Robert Haas wrote:
> > I propose to remove this open item from
> > https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
> >
> > On the original thread (BUG #17257), Alexander Lakhin says that he
> > can't reproduce this after dad1539ae/18b87b201. Based on my analysis
>
> I have observed the infinite loop in production with v15.5, so that
> non-reproduce outcome is a limitation in the test procedure. (v14.2 added
> those two commits.)
How closely have you analyzed those production occurences? It's not too hard
to imagine some form of corruption that leads to such a loop, but which isn't
related to the horizon going backwards? E.g. a corrupted HOT chain can lead
to heap_page_prune() not acting on a DEAD tuple, but lazy_scan_prune() would
then encounter a DEAD tuple.
> > of the code, I suspect that there is a residual bug, or at least that
> > there was one prior to 6f47f6883151366c031cd6fd4011e66d2c702a90.
>
> Can you say more about how 6f47f6883151366c031cd6fd4011e66d2c702a90 mitigated
> the regression that 1ccc1e05ae introduced? Thanks for discovering that.
Which regression has 1ccc1e05ae actually introduced? As I pointed out
upthread, the proposed path to corruption doesn't seem to actually lead to
corruption, "just" an error? Which actually seems considerably better than an
endless retry loop that cannot be cancelled.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-15 23:08 Michael Paquier <michael@paquier.xyz>
parent: Robert Haas <robertmhaas@gmail.com>
2 siblings, 0 replies; 62+ messages in thread
From: Michael Paquier @ 2024-04-15 23:08 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Mon, Apr 15, 2024 at 12:35:59PM -0400, Robert Haas wrote:
> I propose to remove this open item from
> https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
Rather than remove this item and take the risk to forget about it, I'd
recommend to move it to the section "Older bugs affecting stable
branches" for "Live issues".
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-16 03:58 Noah Misch <noah@leadboat.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-04-16 03:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Apr 15, 2024 at 02:10:20PM -0700, Andres Freund wrote:
> On 2024-04-15 13:52:04 -0700, Noah Misch wrote:
> > On Mon, Apr 15, 2024 at 12:35:59PM -0400, Robert Haas wrote:
> > > I propose to remove this open item from
> > > https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
> > >
> > > On the original thread (BUG #17257), Alexander Lakhin says that he
> > > can't reproduce this after dad1539ae/18b87b201. Based on my analysis
> >
> > I have observed the infinite loop in production with v15.5, so that
> > non-reproduce outcome is a limitation in the test procedure. (v14.2 added
> > those two commits.)
>
> How closely have you analyzed those production occurences? It's not too hard
> to imagine some form of corruption that leads to such a loop, but which isn't
> related to the horizon going backwards? E.g. a corrupted HOT chain can lead
> to heap_page_prune() not acting on a DEAD tuple, but lazy_scan_prune() would
> then encounter a DEAD tuple.
One occurrence had these facts:
HeapTupleHeaderGetXmin = 95271613
HeapTupleHeaderGetUpdateXid = 95280147
vacrel->OldestXmin = 95317451
vacrel->vistest->definitely_needed = 95318928
vacrel->vistest->maybe_needed = 93624425
How compatible are those with the corruption vectors you have in view?
> > > of the code, I suspect that there is a residual bug, or at least that
> > > there was one prior to 6f47f6883151366c031cd6fd4011e66d2c702a90.
> >
> > Can you say more about how 6f47f6883151366c031cd6fd4011e66d2c702a90 mitigated
> > the regression that 1ccc1e05ae introduced? Thanks for discovering that.
>
> Which regression has 1ccc1e05ae actually introduced? As I pointed out
> upthread, the proposed path to corruption doesn't seem to actually lead to
> corruption, "just" an error? Which actually seems considerably better than an
> endless retry loop that cannot be cancelled.
A transient, spurious error is far better than an uninterruptible infinite
loop with a buffer content lock held. If a transient error is the consistent
outcome, I would agree 1ccc1e05ae improved the situation and didn't regress
it. That would close the open item. I tried briefly to understand
https://postgr.es/m/flat/20240415173913.4zyyrwaftujxthf2@awork3.anarazel.de
but I felt verifying its argument was going to be a big job for me. Would
those errors happen transiently, like the infinite loop, or would they persist
until something resets the tuple fields (e.g. ATRewriteTables())?
Thanks,
nm
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-16 18:01 Andres Freund <andres@anarazel.de>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 62+ messages in thread
From: Andres Freund @ 2024-04-16 18:01 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-04-15 20:58:25 -0700, Noah Misch wrote:
> On Mon, Apr 15, 2024 at 02:10:20PM -0700, Andres Freund wrote:
> > On 2024-04-15 13:52:04 -0700, Noah Misch wrote:
> > > On Mon, Apr 15, 2024 at 12:35:59PM -0400, Robert Haas wrote:
> > > > I propose to remove this open item from
> > > > https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items
> > > >
> > > > On the original thread (BUG #17257), Alexander Lakhin says that he
> > > > can't reproduce this after dad1539ae/18b87b201. Based on my analysis
> > >
> > > I have observed the infinite loop in production with v15.5, so that
> > > non-reproduce outcome is a limitation in the test procedure. (v14.2 added
> > > those two commits.)
> >
> > How closely have you analyzed those production occurences? It's not too hard
> > to imagine some form of corruption that leads to such a loop, but which isn't
> > related to the horizon going backwards? E.g. a corrupted HOT chain can lead
> > to heap_page_prune() not acting on a DEAD tuple, but lazy_scan_prune() would
> > then encounter a DEAD tuple.
>
> One occurrence had these facts:
>
> HeapTupleHeaderGetXmin = 95271613
> HeapTupleHeaderGetUpdateXid = 95280147
> vacrel->OldestXmin = 95317451
> vacrel->vistest->definitely_needed = 95318928
> vacrel->vistest->maybe_needed = 93624425
>
> How compatible are those with the corruption vectors you have in view?
Do you have more information about the page this was on? E.g. pageinspect
output? Or at least the infomasks of that tuple? I assume this was a normal
data table (i.e. not a [shared|user] catalog table or temp table)?
Do you know what ComputeXidHorizonsResultLastXmin, RecentXmin were set to?
> > > > of the code, I suspect that there is a residual bug, or at least that
> > > > there was one prior to 6f47f6883151366c031cd6fd4011e66d2c702a90.
> > >
> > > Can you say more about how 6f47f6883151366c031cd6fd4011e66d2c702a90 mitigated
> > > the regression that 1ccc1e05ae introduced? Thanks for discovering that.
> >
> > Which regression has 1ccc1e05ae actually introduced? As I pointed out
> > upthread, the proposed path to corruption doesn't seem to actually lead to
> > corruption, "just" an error? Which actually seems considerably better than an
> > endless retry loop that cannot be cancelled.
>
> A transient, spurious error is far better than an uninterruptible infinite
> loop with a buffer content lock held. If a transient error is the consistent
> outcome, I would agree 1ccc1e05ae improved the situation and didn't regress
> it.
I don't think an error is a guaranteed outcome - if there's no conflict
between the tuple's xids and the visibility cutoff, we'd just continue without
a problem. Bt that'd not be a correctness issue, we'd just not be able to
advance relfrozenxid as far as we'd like. But in cases where the xids are
before the cutoffs, yes, an error would occur.
> I tried briefly to understand
> https://postgr.es/m/flat/20240415173913.4zyyrwaftujxthf2@awork3.anarazel.de
> but I felt verifying its argument was going to be a big job for me. Would
> those errors happen transiently, like the infinite loop, or would they
> persist until something resets the tuple fields (e.g. ATRewriteTables())?
I think they'd be transient, because the visibility information during the
next vacuum would presumably not be "skewed" anymore? Of course it's possible
you'd re-encounter the problem, if you constantly have horizons going back and
forth. But I'd still classify that as transient.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-16 19:34 Noah Misch <noah@leadboat.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 62+ messages in thread
From: Noah Misch @ 2024-04-16 19:34 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Apr 16, 2024 at 11:01:08AM -0700, Andres Freund wrote:
> On 2024-04-15 20:58:25 -0700, Noah Misch wrote:
> > On Mon, Apr 15, 2024 at 02:10:20PM -0700, Andres Freund wrote:
> > > On 2024-04-15 13:52:04 -0700, Noah Misch wrote:
> > > > I have observed the infinite loop in production with v15.5, so that
> > > > non-reproduce outcome is a limitation in the test procedure. (v14.2 added
> > > > those two commits.)
> > >
> > > How closely have you analyzed those production occurences? It's not too hard
> > > to imagine some form of corruption that leads to such a loop, but which isn't
> > > related to the horizon going backwards? E.g. a corrupted HOT chain can lead
> > > to heap_page_prune() not acting on a DEAD tuple, but lazy_scan_prune() would
> > > then encounter a DEAD tuple.
I've not seen this recur for any one table, so I think we can rule out
corruption modes that would reach the loop every time. (If a hypothesized
loop explanation calls for both corruption and horizon movement, that could
still apply.)
> > One occurrence had these facts:
> >
> > HeapTupleHeaderGetXmin = 95271613
> > HeapTupleHeaderGetUpdateXid = 95280147
> > vacrel->OldestXmin = 95317451
> > vacrel->vistest->definitely_needed = 95318928
> > vacrel->vistest->maybe_needed = 93624425
> >
> > How compatible are those with the corruption vectors you have in view?
>
> Do you have more information about the page this was on? E.g. pageinspect
> output? Or at least the infomasks of that tuple?
No, unfortunately.
> I assume this was a normal
> data table (i.e. not a [shared|user] catalog table or temp table)?
Normal data table
> Do you know what ComputeXidHorizonsResultLastXmin, RecentXmin were set to?
No.
> > I tried briefly to understand
> > https://postgr.es/m/flat/20240415173913.4zyyrwaftujxthf2@awork3.anarazel.de
> > but I felt verifying its argument was going to be a big job for me. Would
> > those errors happen transiently, like the infinite loop, or would they
> > persist until something resets the tuple fields (e.g. ATRewriteTables())?
>
> I think they'd be transient, because the visibility information during the
> next vacuum would presumably not be "skewed" anymore?
That is good.
> Of course it's possible
> you'd re-encounter the problem, if you constantly have horizons going back and
> forth. But I'd still classify that as transient.
Certainly.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-04-26 20:46 Melanie Plageman <melanieplageman@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-04-26 20:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Apr 15, 2024 at 2:52 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2024-03-29 12:05:31 -0400, Robert Haas wrote:
> > Perhaps, for some reason I'm not grokking at the moment, preventing
> > maybe_needed from retreating would be good enough to prevent trouble
> > in practice, but it doesn't appear to me to be the most principled
> > solution as of this writing.
>
> If we prevent maybe_needed from retreating below the OldestXmin value used for
> cutoff, then pruning should never be less aggressive than what's needed by
> lazy_scan_prune(). So lazy_scan_prune() shouldn't be able to encounter tuples
> that weren't considered removable in heap_page_prune(), in < 17, nor would
> heap_page_prune_and_freeze() have that issue.
>
> I think one fairly easy fix for this would be to pass OldestXmin to
> heap_page_prune[_and_freeze](), and have heap_prune_satisfies_vacuum() first
> check OldestXmin and only if that considers the tuple still running, use
> GlobalVisTestIsRemovableXid(). That continues to allow us to prune more
> aggressively, without any potential risk of IsRemoval being too conservative.
It seems to me that in all stable versions containing the retry loop
(from 8523492d4e34), the following can theoretically happen, causing
an infinite loop in lazy_scan_prune():
1) tuple's xmin and xmax are older than VacuumCutoffs->OldestXmin
2) heap_page_prune()-> heap_prune_satisfies_vacuum()->
HeapTupleSatisfiesVacuumHorizon() returns HEAPTUPLE_RECENTLY_DEAD
3) in GlobalVisTestIsRemovableXid(), dead_after is between
GlobalVisState->maybe_needed and definitely_needed, causing
GlobalVisState to be recomputed
4) GlobalVisState->maybe_needed moves backwards
5) tuple is not removable because dead_after is now newer than maybe_needed
6) in lazy_scan_prune(), HeapTupleSatisfiesVacuum() returns
HEAPTUPLE_DEAD because dead_after is older than OldestXmin
7) infinite loop
One way to fix this (as mentioned earlier in this thread) is to
backport 1ccc1e05ae because it eliminates the retry loop. In our above
example, lazy_scan_prune() reuses the tuple's HEAPTUPLE_RECENTLY_DEAD
HTSV_Result instead of recomputing it. We'd have to rule out any of
the data corruption claims about that commit to justify this, but I am
not yet convinced that 1ccc1e05ae could cause any problems with
relfrozenxid advancement.
Andres' idea of passing VacuumCutoffs->OldestXmin to heap_page_prune()
makes sense. We could add a member to PruneState, oldest_xmin, and
initialize it to InvalidTransactionId for on-access pruning and to
VacuumCutoffs->OldestXmin for vacuum. Then, in
heap_prune_satisfies_vacuum(), we could add this if branch:
if (TransactionIdPrecedes(dead_after, prstate->oldest_xmin))
to here:
- if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
+
+ if (TransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ res = HEAPTUPLE_DEAD;
+ else if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
res = HEAPTUPLE_DEAD;
else if (OldSnapshotThresholdActive())
This seems like a more targeted fix than backpatching 1ccc1e05ae.
I plan to attempt to write a test case that repros this scenario
(infinite loop on stable branch) next week.
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-09 21:56 Melanie Plageman <melanieplageman@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-05-09 21:56 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Apr 26, 2024 at 4:46 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
>
> On Mon, Apr 15, 2024 at 2:52 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2024-03-29 12:05:31 -0400, Robert Haas wrote:
> > > Perhaps, for some reason I'm not grokking at the moment, preventing
> > > maybe_needed from retreating would be good enough to prevent trouble
> > > in practice, but it doesn't appear to me to be the most principled
> > > solution as of this writing.
> >
> > If we prevent maybe_needed from retreating below the OldestXmin value used for
> > cutoff, then pruning should never be less aggressive than what's needed by
> > lazy_scan_prune(). So lazy_scan_prune() shouldn't be able to encounter tuples
> > that weren't considered removable in heap_page_prune(), in < 17, nor would
> > heap_page_prune_and_freeze() have that issue.
> >
> > I think one fairly easy fix for this would be to pass OldestXmin to
> > heap_page_prune[_and_freeze](), and have heap_prune_satisfies_vacuum() first
> > check OldestXmin and only if that considers the tuple still running, use
> > GlobalVisTestIsRemovableXid(). That continues to allow us to prune more
> > aggressively, without any potential risk of IsRemoval being too conservative.
>
> It seems to me that in all stable versions containing the retry loop
> (from 8523492d4e34), the following can theoretically happen, causing
> an infinite loop in lazy_scan_prune():
>
> 1) tuple's xmin and xmax are older than VacuumCutoffs->OldestXmin
> 2) heap_page_prune()-> heap_prune_satisfies_vacuum()->
> HeapTupleSatisfiesVacuumHorizon() returns HEAPTUPLE_RECENTLY_DEAD
> 3) in GlobalVisTestIsRemovableXid(), dead_after is between
> GlobalVisState->maybe_needed and definitely_needed, causing
> GlobalVisState to be recomputed
> 4) GlobalVisState->maybe_needed moves backwards
> 5) tuple is not removable because dead_after is now newer than maybe_needed
> 6) in lazy_scan_prune(), HeapTupleSatisfiesVacuum() returns
> HEAPTUPLE_DEAD because dead_after is older than OldestXmin
> 7) infinite loop
>
> One way to fix this (as mentioned earlier in this thread) is to
> backport 1ccc1e05ae because it eliminates the retry loop. In our above
> example, lazy_scan_prune() reuses the tuple's HEAPTUPLE_RECENTLY_DEAD
> HTSV_Result instead of recomputing it. We'd have to rule out any of
> the data corruption claims about that commit to justify this, but I am
> not yet convinced that 1ccc1e05ae could cause any problems with
> relfrozenxid advancement.
>
> Andres' idea of passing VacuumCutoffs->OldestXmin to heap_page_prune()
> makes sense. We could add a member to PruneState, oldest_xmin, and
> initialize it to InvalidTransactionId for on-access pruning and to
> VacuumCutoffs->OldestXmin for vacuum. Then, in
> heap_prune_satisfies_vacuum(), we could add this if branch:
>
> if (TransactionIdPrecedes(dead_after, prstate->oldest_xmin))
>
> to here:
>
> - if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
> +
> + if (TransactionIdPrecedes(dead_after, prstate->oldest_xmin))
> + res = HEAPTUPLE_DEAD;
> + else if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
> res = HEAPTUPLE_DEAD;
> else if (OldSnapshotThresholdActive())
>
> This seems like a more targeted fix than backpatching 1ccc1e05ae.
>
> I plan to attempt to write a test case that repros this scenario
> (infinite loop on stable branch) next week.
I can repro the hang on 14 and 15 with the following:
I start a primary with the following options:
wal_level = replica, hot_standby_feedback=on
primary_conninfo='host=localhost port=6432 dbname=postgres'
then take a basebackup and start the replica.
Then I connect with psql five times (sessions A, B, and C on the
primary and sessions A and B on the replica):
Then I do the following steps.
step 1:
PRIMARY SESSION A:
--
CREATE TABLE foo(a INT) WITH (autovacuum_enabled=false);
CREATE INDEX foo_idx ON foo(a);
INSERT INTO foo VALUES (1);
UPDATE foo SET a = 100 WHERE a = 1;
DROP TABLESPACE IF EXISTS foo_tablespace;
CREATE TABLESPACE foo_tablespace LOCATION 'somelocation';
step 2:
REPLICA SESSION A:
--
ALTER SYSTEM SET primary_conninfo = 'host=localhost port=9999
dbname=postgres';
SELECT pg_reload_conf();
SELECT pg_terminate_backend((select pid from pg_stat_activity where
backend_type = 'walreceiver'));
BEGIN;
DECLARE c1 CURSOR FOR SELECT * FROM foo;
FETCH FORWARD FROM c1;
step 3:
PRIMARY SESSION A:
--
INSERT INTO foo VALUES (99);
UPDATE foo SET a = 100 WHERE a = 99;
step 4:
PRIMARY SESSION B:
--
BEGIN;
DECLARE c1 CURSOR FOR SELECT * FROM foo;
FETCH FORWARD FROM c1;
step 5:
PRIMARY SESSION C:
--
BEGIN;
ALTER INDEX foo_idx SET TABLESPACE foo_tablespace;
step 6:
PRIMARY SESSION A:
--
VACUUM (FREEZE) foo;
step 7:
PRIMARY SESSION B:
--
COMMIT;
step 8:
REPLICA SESSION B:
--
ALTER SYSTEM SET primary_conninfo = 'host=localhost port=6432
dbname=postgres';
SELECT pg_reload_conf();
step 9:
PRIMARY SESSION C:
--
COMMIT;
step 10:
REPLICA SESSION A:
--
COMMIT;
This infinitely loops in lazy_scan_prune() on 14 and 15. On 16 and
master, everything is normal (for this specific scenario).
The steps roughly follow what I wrote in my previous email.
The difference between 16/master and 14/15, is that in 14/15,
vacuum_set_xid_limits() is called before opening indexes. This allows
an intervening relcache invalidation caused by modifying an index on
the table being vacuumed to force us to rebuild the catalog snapshot
between vacuum_set_xid_limits() and eventually pruning the tuples.
Rebuilding the catalog snapshot will update RecentXmin to a
potentially different value than ComputeXidHorizonsResultLastXmin (
ComputeXidHorizonResultLastXmin is set to RecentXmin during
vacuum_set_xid_limits()). This provides the opportunity for
GlobalVisTestShouldUpdate() to return true while pruning RECENTLY_DEAD
tuples.
See this line in GlobalVisTestShouldUpdate():
return RecentXmin != ComputeXidHorizonsResultLastXmin;
On 16 and master, AFAICT, RecentXmin will always equal
ComputeXidHorizonsResultLastXmin in GlobalVisTestShouldUpdate() when
called through heap_prune_satisfies_vacuum(). So, even if a wal sender
pushes the horizon backwards on the primary, vacuum's
GlobalVisState->maybe_needed won't be pushed back to a value lower
than OldestXmin.
I haven't yet written the fix and tested it. I want to investigate a
bit more and write more detailed notes on the repro steps. I also want
to fully convince myself this isn't possible on master or 16.
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-13 03:19 Bowen Shi <zxwsbg12138@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-13 03:19 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
Hi,
> Obviously we should actually fix this on back branches, but could we
> at least make the retry loop interruptible in some way so people could
> use pg_cancel/terminate_backend() on a stuck autovacuum worker or
> vacuum process?
If the problem happens in versions <= PG 16, we don't have a good solution
(vacuum process holds the exclusive lock cause checkpoint hangs).
Maybe we can make the retry loop interruptible first. However, since we are
using START_CRIT_SECTION, we cannot simply use CHECK_FOR_INTERRUPTS to
handle it.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-13 14:42 Melanie Plageman <melanieplageman@gmail.com>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-05-13 14:42 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Sun, May 12, 2024 at 11:19 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>
> Hi,
>>
>> Obviously we should actually fix this on back branches, but could we
>> at least make the retry loop interruptible in some way so people could
>> use pg_cancel/terminate_backend() on a stuck autovacuum worker or
>> vacuum process?
>
>
> If the problem happens in versions <= PG 16, we don't have a good solution (vacuum process holds the exclusive lock cause checkpoint hangs).
>
> Maybe we can make the retry loop interruptible first. However, since we are using START_CRIT_SECTION, we cannot simply use CHECK_FOR_INTERRUPTS to handle it.
As far as I can tell, in 14 and 15, the versions where the issue
reported here is present, there is not a critical section in the
section of code looped through in the retry loop in lazy_scan_prune().
We can actually fix the particular issue I reproduced with the
attached patch. However, I think it is still worth making the retry
loop interruptible in case there are other ways to end up infinitely
looping in the retry loop in lazy_scan_prune().
- Melanie
Attachments:
v1-0001-Fix-vacuum-hang.patch (text/x-patch, 4.7K) download
From 33ec54d05441183ba908a728186b2a6751bc4cf5 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 13 May 2024 10:27:50 -0400
Subject: [PATCH v1] Fix vacuum hang
---
src/backend/access/heap/pruneheap.c | 33 ++++++++++++++++++++--------
src/backend/access/heap/vacuumlazy.c | 2 +-
src/include/access/heapam.h | 1 +
3 files changed, 26 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index f7f8056d631..0356391c5db 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -35,6 +35,12 @@ typedef struct
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
+ /*
+ * Used only by vacuum. The oldest xmin at the beginning of vacuuming the
+ * relation.
+ */
+ TransactionId oldest_xmin;
+
/*
* Thresholds set by TransactionIdLimitedForOldSnapshots() if they have
* been computed (done on demand, and only if
@@ -203,6 +209,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
{
/* OK to prune */
(void) heap_page_prune(relation, buffer, vistest,
+ InvalidTransactionId,
limited_xmin, limited_ts,
true, NULL);
}
@@ -218,11 +225,14 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*
* Caller must have pin and buffer cleanup lock on the page.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum). old_snap_xmin / old_snap_ts need to
- * either have been set by TransactionIdLimitedForOldSnapshots, or
- * InvalidTransactionId/0 respectively.
+ * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD (see
+ * heap_prune_satisfies_vacuum and HeapTupleSatisfiesVacuum).
+ *
+ * oldest_xmin is only set to a valid TransactionId by vacuum. It is the oldest
+ * removable TransactionId at the beginning of vacuuming the relation.
+ *
+ * old_snap_xmin / old_snap_ts need to either have been set by
+ * TransactionIdLimitedForOldSnapshots, or InvalidTransactionId/0 respectively.
*
* If report_stats is true then we send the number of reclaimed heap-only
* tuples to pgstats. (This must be false during vacuum, since vacuum will
@@ -237,6 +247,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int
heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
+ TransactionId oldest_xmin,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
bool report_stats,
@@ -263,6 +274,7 @@ heap_page_prune(Relation relation, Buffer buffer,
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
prstate.vistest = vistest;
+ prstate.oldest_xmin = oldest_xmin;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
@@ -512,11 +524,14 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
/*
- * First check if GlobalVisTestIsRemovableXid() is sufficient to find the
- * row dead. If not, and old_snapshot_threshold is enabled, try to use the
- * lowered horizon.
+ * First check if oldest_xmin or GlobalVisTestIsRemovableXid() is
+ * sufficient to find the row dead. If not, and old_snapshot_threshold is
+ * enabled, try to use the lowered horizon.
*/
- if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
+ if (TransactionIdIsValid(prstate->oldest_xmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ res = HEAPTUPLE_DEAD;
+ else if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
res = HEAPTUPLE_DEAD;
else if (OldSnapshotThresholdActive())
{
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8aab6e324e0..31fdc8e6cd9 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1738,7 +1738,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vistest, vacrel->OldestXmin,
InvalidTransactionId, 0, false,
&vacrel->offnum);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 4f1dff9ca1b..780b30fe74c 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -187,6 +187,7 @@ struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern int heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
+ TransactionId oldest_xmin,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
bool report_stats,
--
2.34.1
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-14 03:42 Bowen Shi <zxwsbg12138@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-14 03:42 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Mon, May 13, 2024 at 10:42 PM Melanie Plageman <melanieplageman@gmail.com>
wrote:
> On Sun, May 12, 2024 at 11:19 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
> >
> > Hi,
> >>
> >> Obviously we should actually fix this on back branches, but could we
> >> at least make the retry loop interruptible in some way so people could
> >> use pg_cancel/terminate_backend() on a stuck autovacuum worker or
> >> vacuum process?
> >
> >
> > If the problem happens in versions <= PG 16, we don't have a good
> solution (vacuum process holds the exclusive lock cause checkpoint hangs).
> >
> > Maybe we can make the retry loop interruptible first. However, since we
> are using START_CRIT_SECTION, we cannot simply use CHECK_FOR_INTERRUPTS to
> handle it.
>
> As far as I can tell, in 14 and 15, the versions where the issue
> reported here is present, there is not a critical section in the
> section of code looped through in the retry loop in lazy_scan_prune().
>
You are correct, I tried again to add CHECK_FOR_INTERRUPTS in the retry
loop, and when attempting to interrupt the current loop using the
pg_terminate_backend function, the value of InterruptHoldoffCount is 1,
which causes the vacuum to not end.
We can actually fix the particular issue I reproduced with the
> attached patch. However, I think it is still worth making the retry
> loop interruptible in case there are other ways to end up infinitely
> looping in the retry loop in lazy_scan_prune().
I attempted to apply the patch on the REL_15_STABLE branch, but encountered
some conflicts. May I ask which branch you are using?
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-14 23:33 Melanie Plageman <melanieplageman@gmail.com>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-05-14 23:33 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Mon, May 13, 2024 at 11:42 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>
> On Mon, May 13, 2024 at 10:42 PM Melanie Plageman <melanieplageman@gmail.com> wrote:
>>
>> On Sun, May 12, 2024 at 11:19 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>> >
>> > Hi,
>> >>
>> >> Obviously we should actually fix this on back branches, but could we
>> >> at least make the retry loop interruptible in some way so people could
>> >> use pg_cancel/terminate_backend() on a stuck autovacuum worker or
>> >> vacuum process?
>> >
>> >
>> > If the problem happens in versions <= PG 16, we don't have a good solution (vacuum process holds the exclusive lock cause checkpoint hangs).
>> >
>> > Maybe we can make the retry loop interruptible first. However, since we are using START_CRIT_SECTION, we cannot simply use CHECK_FOR_INTERRUPTS to handle it.
>>
>> As far as I can tell, in 14 and 15, the versions where the issue
>> reported here is present, there is not a critical section in the
>> section of code looped through in the retry loop in lazy_scan_prune().
>
>
> You are correct, I tried again to add CHECK_FOR_INTERRUPTS in the retry loop, and when attempting to interrupt the current loop using the pg_terminate_backend function, the value of InterruptHoldoffCount is 1, which causes the vacuum to not end.
Yes, great point. Actually, Andres and I discussed this today
off-list, and he reminded me that because vacuum is holding a content
lock on the page here, InterruptHoldoffCount will be at least 1. We
could RESUME_INTERRUPTS() but we probably don't want to process
interrupts while holding the page lock here if we don't do it in other
places. And it's hard to say under what conditions we would want to
drop the page lock here.
Are you reproducing the hang locally with my repro? Or do you have
your own repro? How are you testing pg_terminate_backend() and seeing
that the InterruptHoldoffCount is 1?
>> We can actually fix the particular issue I reproduced with the
>> attached patch. However, I think it is still worth making the retry
>> loop interruptible in case there are other ways to end up infinitely
>> looping in the retry loop in lazy_scan_prune().
>
>
> I attempted to apply the patch on the REL_15_STABLE branch, but encountered some conflicts. May I ask which branch you are using?
Sorry, I should have mentioned that patch was against REL_14_STABLE.
Attached patch has the same functionality but should apply cleanly
against REL_15_STABLE.
- Melanie
Attachments:
fix_hang_15.patch (text/x-patch, 4.9K) download
From eb5ed39ee00430755c45e4ed55ba1cde4c6fd79f Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 14 May 2024 10:35:14 -0400
Subject: [PATCH v1] fix
---
src/backend/access/heap/pruneheap.c | 36 +++++++++++++++++++---------
src/backend/access/heap/vacuumlazy.c | 2 +-
src/include/access/heapam.h | 1 +
3 files changed, 27 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 9f43bbe25f5..65d409e8b12 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -36,6 +36,12 @@ typedef struct
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
+ /*
+ * Used only by vacuum. The oldest xmin at the beginning of vacuuming the
+ * relation.
+ */
+ TransactionId oldest_xmin;
+
/*
* Thresholds set by TransactionIdLimitedForOldSnapshots() if they have
* been computed (done on demand, and only if
@@ -206,8 +212,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int ndeleted,
nnewlpdead;
- ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
- limited_ts, &nnewlpdead, NULL);
+ ndeleted = heap_page_prune(relation, buffer, vistest, InvalidTransactionId,
+ limited_xmin, limited_ts, &nnewlpdead, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -248,11 +254,14 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* also need to account for a reduction in the length of the line pointer
* array following array truncation by us.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum). old_snap_xmin / old_snap_ts need to
- * either have been set by TransactionIdLimitedForOldSnapshots, or
- * InvalidTransactionId/0 respectively.
+ * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD (see
+ * heap_prune_satisfies_vacuum and HeapTupleSatisfiesVacuum).
+ *
+ * oldest_xmin is only set to a valid TransactionId by vacuum. It is the oldest
+ * removable TransactionId at the beginning of vacuuming the relation.
+ *
+ * old_snap_xmin / old_snap_ts need to either have been set by
+ * TransactionIdLimitedForOldSnapshots, or InvalidTransactionId/0 respectively.
*
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
@@ -265,6 +274,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int
heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
+ TransactionId oldest_xmin,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
int *nnewlpdead,
@@ -292,6 +302,7 @@ heap_page_prune(Relation relation, Buffer buffer,
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
prstate.vistest = vistest;
+ prstate.oldest_xmin = oldest_xmin;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
@@ -520,11 +531,14 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
/*
- * First check if GlobalVisTestIsRemovableXid() is sufficient to find the
- * row dead. If not, and old_snapshot_threshold is enabled, try to use the
- * lowered horizon.
+ * First check if oldest_xmin or GlobalVisTestIsRemovableXid() is
+ * sufficient to find the row dead. If not, and old_snapshot_threshold is
+ * enabled, try to use the lowered horizon.
*/
- if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
+ if (TransactionIdIsValid(prstate->oldest_xmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ res = HEAPTUPLE_DEAD;
+ else if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
res = HEAPTUPLE_DEAD;
else if (OldSnapshotThresholdActive())
{
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2e61e2f20b4..a027458cf69 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1585,7 +1585,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, vacrel->OldestXmin,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index abf62d9df79..e39fa14c1b2 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -186,6 +186,7 @@ struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern int heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
+ TransactionId oldest_xmin,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
int *nnewlpdead,
--
2.34.1
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-15 02:43 Bowen Shi <zxwsbg12138@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-15 02:43 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
> On Wed, May 15, 2024 at 7:33 AM Melanie Plageman <
> melanieplageman@gmail.com> wrote:
> Are you reproducing the hang locally with my repro? Or do you have
> your own repro? How are you testing pg_terminate_backend() and seeing
> that the InterruptHoldoffCount is 1?
>
I noticed this problem occurring several times on PG16, but I didn't use
your repro, and haven't found a way to reproduce the problem. Instead, I
simply force the vacuum to enter a retry loop with the following code and
found InterruptHoldoffCount is 1.
// if (unlikely(res == HEAPTUPLE_DEAD))
goto retry;
Sorry, I should have mentioned that patch was against REL_14_STABLE.
> Attached patch has the same functionality but should apply cleanly
> against REL_15_STABLE.
>
Thanks, I would try that.
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-15 13:26 Melanie Plageman <melanieplageman@gmail.com>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-05-15 13:26 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Tue, May 14, 2024 at 10:43 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>
>
>> On Wed, May 15, 2024 at 7:33 AM Melanie Plageman <melanieplageman@gmail.com> wrote:
>>
>>
>> Are you reproducing the hang locally with my repro? Or do you have
>> your own repro? How are you testing pg_terminate_backend() and seeing
>> that the InterruptHoldoffCount is 1?
>
>
> I noticed this problem occurring several times on PG16, but I didn't use your repro, and haven't found a way to reproduce the problem. Instead, I simply force the vacuum to enter a retry loop with the following code and found InterruptHoldoffCount is 1.
You saw this problem on PG16? Did you see vacuum hanging or you saw
some kind of incorrect relfrozenxid advancement? My repro does not
work for PG16, and I do not see how we could face this problem on
PG16. It would be very helpful if you have any more details.
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 02:21 Bowen Shi <zxwsbg12138@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-16 02:21 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
On Wed, May 15, 2024 at 9:27 PM Melanie Plageman <melanieplageman@gmail.com>
wrote:
> You saw this problem on PG16? Did you see vacuum hanging or you saw
> some kind of incorrect relfrozenxid advancement? My repro does not
> work for PG16, and I do not see how we could face this problem on
> PG16. It would be very helpful if you have any more details.
>
Yes, I found the vacuum process hanging in a retry loop on PG16. I think
it was the problem discussed in this thread. Here is some information:
vacrel: OldestXmin = 1106, relfrozenxid = 927
vacrel->vistest: maybe_needed = 1079, definitely_need = 1106
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 03:33 Andres Freund <andres@anarazel.de>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Andres Freund @ 2024-05-16 03:33 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-05-16 10:21:59 +0800, Bowen Shi wrote:
> Yes, I found the vacuum process hanging in a retry loop on PG16. I think
> it was the problem discussed in this thread. Here is some information:
Very interesting.
Do you have the page contents that triggered the issue?
> vacrel: OldestXmin = 1106, relfrozenxid = 927
> vacrel->vistest: maybe_needed = 1079, definitely_need = 1106
Do you have any other variables? E.g. RecentXmin,
ComputeXidHorizonsResultLastXmin?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 07:31 Bowen Shi <zxwsbg12138@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-16 07:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On Thu, May 16, 2024 at 11:33 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2024-05-16 10:21:59 +0800, Bowen Shi wrote:
> > Yes, I found the vacuum process hanging in a retry loop on PG16. I think
> > it was the problem discussed in this thread. Here is some information:
>
> Very interesting.
>
>
> Do you have the page contents that triggered the issue?
Yes, I have preserved the coredump file of vacuum and several other related
processes. It has been at least 10 minutes stuck in the retry loop since I
dumped this core file.
(gdb) p *(BufferDesc *) (&BufferDescriptors[(buffer)].bufferdesc)
$8 = {tag = {spcOid = 1663, dbOid = 16425, relNumber = 61755, forkNum =
MAIN_FORKNUM,
blockNum = 50002}, buf_id = 67521, state = {value = 3548643328},
wait_backend_pgprocno = 2147483647, freeNext = -2, content_lock =
{tranche = 78, state = {
value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}}
And in lazy_scan_prune() :
(gdb) p maxoff
$11 = 32
(gdb) p offnum
$12 = 1
Item info for offsets 1 to 32 using loop:
(gdb) define print_itemids
> set $i = 1
> while $i <= 32
> printf "offset %d: ", $i
> print ((PageHeader)(page))->pd_linp[$i - 1]
> set $i = $i + 1
> end
> end
(gdb) print_itemids
offset 1: $1 = {lp_off = 7944, lp_flags = 1, lp_len = 248}
offset 2: $2 = {lp_off = 7696, lp_flags = 1, lp_len = 248}
offset 3: $3 = {lp_off = 7448, lp_flags = 1, lp_len = 248}
offset 4: $4 = {lp_off = 7200, lp_flags = 1, lp_len = 248}
offset 5: $5 = {lp_off = 6952, lp_flags = 1, lp_len = 248}
offset 6: $6 = {lp_off = 6704, lp_flags = 1, lp_len = 248}
offset 7: $7 = {lp_off = 6456, lp_flags = 1, lp_len = 248}
offset 8: $8 = {lp_off = 6208, lp_flags = 1, lp_len = 248}
offset 9: $9 = {lp_off = 5960, lp_flags = 1, lp_len = 248}
offset 10: $10 = {lp_off = 5712, lp_flags = 1, lp_len = 248}
offset 11: $11 = {lp_off = 5464, lp_flags = 1, lp_len = 248}
offset 12: $12 = {lp_off = 5216, lp_flags = 1, lp_len = 248}
offset 13: $13 = {lp_off = 4968, lp_flags = 1, lp_len = 248}
offset 14: $14 = {lp_off = 4720, lp_flags = 1, lp_len = 248}
offset 15: $15 = {lp_off = 4472, lp_flags = 1, lp_len = 248}
offset 16: $16 = {lp_off = 4224, lp_flags = 1, lp_len = 248}
offset 17: $17 = {lp_off = 3976, lp_flags = 1, lp_len = 248}
offset 18: $18 = {lp_off = 3728, lp_flags = 1, lp_len = 248}
offset 19: $19 = {lp_off = 3480, lp_flags = 1, lp_len = 248}
offset 20: $20 = {lp_off = 3232, lp_flags = 1, lp_len = 248}
offset 21: $21 = {lp_off = 2984, lp_flags = 1, lp_len = 248}
offset 22: $22 = {lp_off = 2736, lp_flags = 1, lp_len = 248}
offset 23: $23 = {lp_off = 2488, lp_flags = 1, lp_len = 248}
offset 24: $24 = {lp_off = 2240, lp_flags = 1, lp_len = 248}
offset 25: $25 = {lp_off = 1992, lp_flags = 1, lp_len = 248}
offset 26: $26 = {lp_off = 1744, lp_flags = 1, lp_len = 248}
offset 27: $27 = {lp_off = 1496, lp_flags = 1, lp_len = 248}
offset 28: $28 = {lp_off = 1248, lp_flags = 1, lp_len = 248}
offset 29: $29 = {lp_off = 1000, lp_flags = 1, lp_len = 248}
offset 30: $30 = {lp_off = 752, lp_flags = 1, lp_len = 248}
offset 31: $31 = {lp_off = 504, lp_flags = 1, lp_len = 248}
offset 32: $32 = {lp_off = 256, lp_flags = 1, lp_len = 248}
Using following function to print HeapTupleHeader info:
(gdb) define print_all_heaptupleheaders
>set $i = 1
>while $i <= 32
>set $tup = (HeapTupleHeader) ((char *) (page) + ((PageHeader)
(page))->pd_linp[$i - 1].lp_off)
>print *$tup
>set $i = $i + 1
>end
>end
(gdb) print_all_heaptupleheaders
offset 1: $1 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 7},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729f1f ""}
offset 2: $2 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 8},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729e27 ""}
offset 3: $3 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 9},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729d2f ""}
offset 4: $4 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 10},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729c37 ""}
offset 5: $5 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 11},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729b3f ""}
offset 6: $6 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 12},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729a47 ""}
offset 7: $7 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 13},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b72994f ""}
offset 8: $8 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
---Type <return> to continue, or q <return> to quit---
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 14},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729857 ""}
offset 9: $9 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 15},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b72975f ""}
offset 10: $10 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 16},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729667 ""}
offset 11: $11 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 17},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b72956f ""}
offset 12: $12 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 18},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729477 ""}
offset 13: $13 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 19},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b72937f ""}
offset 14: $14 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 20},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729287 ""}
offset 15: $15 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 21},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b72918f ""}
---Type <return> to continue, or q <return> to quit---
offset 16: $16 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 22},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b729097 ""}
offset 17: $17 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 23},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728f9f ""}
offset 18: $18 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 24},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728ea7 ""}
offset 19: $19 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 25},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728daf ""}
offset 20: $20 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 26},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728cb7 ""}
offset 21: $21 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 27},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728bbf ""}
offset 22: $22 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 28},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728ac7 ""}
offset 23: $23 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
---Type <return> to continue, or q <return> to quit---
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 29},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7289cf ""}
offset 24: $24 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 30},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7288d7 ""}
offset 25: $25 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 31},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7287df ""}
offset 26: $26 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54842}, ip_posid = 32},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7286e7 ""}
offset 27: $27 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54843}, ip_posid = 1},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7285ef ""}
offset 28: $28 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54843}, ip_posid = 2},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7284f7 ""}
offset 29: $29 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54843}, ip_posid = 3},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b7283ff ""}
offset 30: $30 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54843}, ip_posid = 4},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728307 ""}
---Type <return> to continue, or q <return> to quit---
offset 31: $31 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54843}, ip_posid = 5},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b72820f ""}
offset 32: $32 = {t_choice = {t_heap = {t_xmin = 1094, t_xmax = 1094,
t_field3 = {t_cid = 5986,
t_xvac = 5986}}, t_datum = {datum_len_ = 1094, datum_typmod = 1094,
datum_typeid = 5986}}, t_ctid = {ip_blkid = {bi_hi = 6, bi_lo =
54843}, ip_posid = 6},
t_infomask2 = 24, t_infomask = 9506, t_hoff = 24 '\030', t_bits =
0x7fa25b728117 ""}
Do you have any other variables? E.g. RecentXmin,
> ComputeXidHorizonsResultLastXmin?
RecentXmin = 1079
ComputeXidHorizonsResultLastXmin = 1079
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 16:38 Andres Freund <andres@anarazel.de>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 2 replies; 62+ messages in thread
From: Andres Freund @ 2024-05-16 16:38 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-05-16 15:31:20 +0800, Bowen Shi wrote:
> Yes, I have preserved the coredump file of vacuum and several other related
> processes. It has been at least 10 minutes stuck in the retry loop since I
> dumped this core file.
Could you print out *vacrel?
I'm wondering if there was index processing, due to the number of tuples. And
if so, what type of indexes. There'd need to be something that could lead to
new snapshots being acquired...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 16:49 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 62+ messages in thread
From: Peter Geoghegan @ 2024-05-16 16:49 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, May 16, 2024 at 12:38 PM Andres Freund <andres@anarazel.de> wrote:
> I'm wondering if there was index processing, due to the number of tuples. And
> if so, what type of indexes. There'd need to be something that could lead to
> new snapshots being acquired...
Did you ever see this theory of mine, about B-Tree page deletion +
recycling? See:
https://www.postgresql.org/message-id/flat/CAH2-Wz%3DzLcnZO8MqPXQLqOLY%3DCAwQhdvs5Ncg6qMb5nMAam0EA%4...
(And related nearby emails from me.)
It looked very much like index vacuuming was involved in some way when
I actually had the opportunity to use gdb against an affected
production instance that ran into the problem.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 19:39 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 2 replies; 62+ messages in thread
From: Andres Freund @ 2024-05-16 19:39 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-05-16 12:49:00 -0400, Peter Geoghegan wrote:
> On Thu, May 16, 2024 at 12:38 PM Andres Freund <andres@anarazel.de> wrote:
> > I'm wondering if there was index processing, due to the number of tuples. And
> > if so, what type of indexes. There'd need to be something that could lead to
> > new snapshots being acquired...
>
> Did you ever see this theory of mine, about B-Tree page deletion +
> recycling? See:
>
> https://www.postgresql.org/message-id/flat/CAH2-Wz%3DzLcnZO8MqPXQLqOLY%3DCAwQhdvs5Ncg6qMb5nMAam0EA%4...
>
> (And related nearby emails from me.)
> It looked very much like index vacuuming was involved in some way when
> I actually had the opportunity to use gdb against an affected
> production instance that ran into the problem.
Hm, were the cases you observed that way using parallel vacuuming? And what
index types were involved?
Melanies reproducer works because there are catalog accesses that can trigger
a recomputation of fuzzy horizon. For testing the "easy" window for that is
the vac_open_indexes() < 16, because it happens after determining the horizon,
but before actually vacuuming.
Now I wonder if there is some codepath triggering catalog lookups during bulk
delete.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 20:03 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 62+ messages in thread
From: Andres Freund @ 2024-05-16 20:03 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-05-16 12:39:53 -0700, Andres Freund wrote:
> On 2024-05-16 12:49:00 -0400, Peter Geoghegan wrote:
> > It looked very much like index vacuuming was involved in some way when
> > I actually had the opportunity to use gdb against an affected
> > production instance that ran into the problem.
>
> Hm, were the cases you observed that way using parallel vacuuming? And what
> index types were involved?
>
> Melanies reproducer works because there are catalog accesses that can trigger
> a recomputation of fuzzy horizon. For testing the "easy" window for that is
> the vac_open_indexes() < 16, because it happens after determining the horizon,
> but before actually vacuuming.
>
> Now I wonder if there is some codepath triggering catalog lookups during bulk
> delete.
There is in gin:
2024-05-16 13:02:56.114 PDT [1632692][client backend][2/4:0][psql] CONTEXT: while scanning block 33 of relation "public.foo"
2024-05-16 13:02:56.114 PDT [1632692][client backend][2/4:0][psql] STATEMENT: VACUUM (VERBOSE, PARALLEL 0) foo;
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
2024-05-16 13:02:56.116 PDT [1632261][postmaster][:0][] DEBUG: server process (PID 1632484) exited with exit code 0
2024-05-16 13:02:56.140 PDT [1632692][client backend][2/4:0][psql] DEBUG: scanned index "foo_gin" to remove 7458 row versions
#0 AcceptInvalidationMessages () at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/inval.c:808
#1 0x0000000000ce1342 in LockRelationOid (relid=1255, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:136
#2 0x000000000081b718 in relation_open (relationId=1255, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/common/relation.c:55
#3 0x00000000008d8116 in table_open (relationId=1255, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/table/table.c:44
#4 0x0000000000ec8b04 in SearchCatCacheMiss (cache=0x289f180, nkeys=1, hashValue=3157698538, hashIndex=106, v1=16641, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1468
#5 0x0000000000ec8a80 in SearchCatCacheInternal (cache=0x289f180, nkeys=1, v1=16641, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1413
#6 0x0000000000ec8773 in SearchCatCache1 (cache=0x289f180, v1=16641) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1281
#7 0x0000000000ee59cf in SearchSysCache1 (cacheId=45, key1=16641) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:225
#8 0x0000000000ef6496 in fmgr_info_cxt_security (functionId=16641, finfo=0x28e7dd8, mcxt=0x28fa5b0, ignore_security=false)
at ../../../../../home/andres/src/postgresql/src/backend/utils/fmgr/fmgr.c:181
#9 0x0000000000ef63cb in fmgr_info_cxt (functionId=16641, finfo=0x28e7dd8, mcxt=0x28fa5b0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/fmgr/fmgr.c:139
#10 0x0000000000894ee7 in index_getprocinfo (irel=0x7f53fae86e60, attnum=1, procnum=2)
at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:902
#11 0x000000000083d962 in initGinState (state=0x7ffc7c077960, index=0x7f53fae86e60)
at ../../../../../home/andres/src/postgresql/src/backend/access/gin/ginutil.c:155
#12 0x000000000083ff1c in ginbulkdelete (info=0x7ffc7c07a010, stats=0x0, callback=0xa559ca <vac_tid_reaped>, callback_state=0x29036d8)
at ../../../../../home/andres/src/postgresql/src/backend/access/gin/ginvacuum.c:582
#13 0x00000000008949bd in index_bulk_delete (info=0x7ffc7c07a010, istat=0x0, callback=0xa559ca <vac_tid_reaped>, callback_state=0x29036d8)
at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:758
#14 0x0000000000a55872 in vac_bulkdel_one_index (ivinfo=0x7ffc7c07a010, istat=0x0, dead_items=0x29036d8, dead_items_info=0x29036b8)
at ../../../../../home/andres/src/postgresql/src/backend/commands/vacuum.c:2498
#15 0x000000000088e94c in lazy_vacuum_one_index (indrel=0x7f53fae86e60, istat=0x0, reltuples=40000, vacrel=0x2902770)
at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:2451
#16 0x000000000088dc94 in lazy_vacuum_all_indexes (vacrel=0x2902770) at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:2032
#17 0x000000000088d9e4 in lazy_vacuum (vacrel=0x2902770) at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:1946
#18 0x000000000088c2e9 in lazy_scan_heap (vacrel=0x2902770) at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:897
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 20:13 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 62+ messages in thread
From: Peter Geoghegan @ 2024-05-16 20:13 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, May 16, 2024 at 3:39 PM Andres Freund <andres@anarazel.de> wrote:
> Hm, were the cases you observed that way using parallel vacuuming? And what
> index types were involved?
They definitely didn't use parallel VACUUM, since autovacuum was
involved in all cases. Only B-Tree indexes were involved. There were
plenty of index page deletions involved, since the underlying workload
involved bulk deletions. Deletions that seemed to leave every heap
page without any heap tuples.
Partitioning also happened to be involved. No particular reason to
suspect that that matters, though.
> Melanies reproducer works because there are catalog accesses that can trigger
> a recomputation of fuzzy horizon. For testing the "easy" window for that is
> the vac_open_indexes() < 16, because it happens after determining the horizon,
> but before actually vacuuming.
What about the call to GetOldestNonRemovableTransactionId() that takes
place in _bt_pendingfsm_finalize()?
> Now I wonder if there is some codepath triggering catalog lookups during bulk
> delete.
I don't think that there's any rule that says that VACUUM cannot do
catalog lookups during bulk deletions. B-Tree page deletion needs to
generate an insertion scan key, so that it can "refind" a page
undergoing deletion. That might require catalog lookups.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 20:29 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 62+ messages in thread
From: Andres Freund @ 2024-05-16 20:29 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-05-16 16:13:35 -0400, Peter Geoghegan wrote:
> On Thu, May 16, 2024 at 3:39 PM Andres Freund <andres@anarazel.de> wrote:
> > Melanies reproducer works because there are catalog accesses that can trigger
> > a recomputation of fuzzy horizon. For testing the "easy" window for that is
> > the vac_open_indexes() < 16, because it happens after determining the horizon,
> > but before actually vacuuming.
>
> What about the call to GetOldestNonRemovableTransactionId() that takes
> place in _bt_pendingfsm_finalize()?
Ah, good catch! That'd do it.
> > Now I wonder if there is some codepath triggering catalog lookups during bulk
> > delete.
>
> I don't think that there's any rule that says that VACUUM cannot do
> catalog lookups during bulk deletions. B-Tree page deletion needs to
> generate an insertion scan key, so that it can "refind" a page
> undergoing deletion. That might require catalog lookups.
I'm not saying there's a hard rule against it. Just that there wasn't an
immediately apparent, nor immediately observable, path for it. As I didn't see
the path to the horizon recomputation, I didn't know how a btbulkdelete in the
middle of the scan would potentially trigger the problem.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 20:37 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 62+ messages in thread
From: Peter Geoghegan @ 2024-05-16 20:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, May 16, 2024 at 4:29 PM Andres Freund <andres@anarazel.de> wrote:
> On 2024-05-16 16:13:35 -0400, Peter Geoghegan wrote:
> > On Thu, May 16, 2024 at 3:39 PM Andres Freund <andres@anarazel.de> wrote:
> > > Melanies reproducer works because there are catalog accesses that can trigger
> > > a recomputation of fuzzy horizon. For testing the "easy" window for that is
> > > the vac_open_indexes() < 16, because it happens after determining the horizon,
> > > but before actually vacuuming.
> >
> > What about the call to GetOldestNonRemovableTransactionId() that takes
> > place in _bt_pendingfsm_finalize()?
>
> Ah, good catch! That'd do it.
That was definitely what happened in the problem cases I saw -- plenty
of B-Tree page deletions. Plus the heap page that lazy_scan_prune
actually got stuck on happened to be the first one after the first
round of bulk deletes of index tuples -- *exactly* the first heap page
scanned. Not just on one occasion -- there were several affected
instances that I had access to at various points, that all looked like
this (same workload for all of them, though). These were 14 and 15
instances (no 16 instances, likely just because 16 wasn't even really
available at the time).
It seems just about impossible that these details were all coincidental.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 20:38 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 62+ messages in thread
From: Andres Freund @ 2024-05-16 20:38 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2024-05-16 13:29:49 -0700, Andres Freund wrote:
> On 2024-05-16 16:13:35 -0400, Peter Geoghegan wrote:
> > > Now I wonder if there is some codepath triggering catalog lookups during bulk
> > > delete.
> >
> > I don't think that there's any rule that says that VACUUM cannot do
> > catalog lookups during bulk deletions. B-Tree page deletion needs to
> > generate an insertion scan key, so that it can "refind" a page
> > undergoing deletion. That might require catalog lookups.
>
> I'm not saying there's a hard rule against it. Just that there wasn't an
> immediately apparent, nor immediately observable, path for it. As I didn't see
> the path to the horizon recomputation, I didn't know how a btbulkdelete in the
> middle of the scan would potentially trigger the problem.
Hm. Actually. I think it might not be correct to do catalog lookups at that
point. But it's a bigger issue than just catalog lookups during bulkdelete:
Once we've done
MyProc->statusFlags |= PROC_IN_VACUUM;
the current backend's snapshots don't prevent rows from being removed
anymore.
I first wrote:
> That's not a huge issue for the pg_class entry itself, as the locks should
> prevent it from being updated. But there are a lot of catalog lookups that
> aren't protected by locks, just normal snapshot semantics.
but as it turns out we haven't even locked the relation at the point we set
PROC_IN_VACUUM.
That seems quite broken.
WRT bulkdelete, there's this comment where we set PROC_IN_VACUUM:
* In lazy vacuum, we can set the PROC_IN_VACUUM flag, which lets
* other concurrent VACUUMs know that they can ignore this one while
* determining their OldestXmin. (The reason we don't set it during a
* full VACUUM is exactly that we may have to run user-defined
* functions for functional indexes, and we want to make sure that if
* they use the snapshot set above, any tuples it requires can't get
* removed from other tables. An index function that depends on the
* contents of other tables is arguably broken, but we won't break it
* here by violating transaction semantics.)
the parenthetical explains that/why we can't evaluate user defined
functions. Which seems to be violated by doing key comparisons, no?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-16 23:57 Melanie Plageman <melanieplageman@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-05-16 23:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, May 9, 2024 at 5:56 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
>
> I can repro the hang on 14 and 15 with the following:
-- snip --
I finally managed to write a TAP test which does the same thing as my
repro. The attached patch contains a test that is meant purely for
demonstration and reproduction by others -- not as a suggestion to add
to the recovery test suite on an ongoing basis.
I also attached the fix (targeting REL_15_STABLE) so that you can try
the repro TAP test with and without the fix.
The test is a bit fragile, so I won't be surprised if it doesn't work
for everyone.
Thank you so much to Andres, Thomas, and Daniel, all of whom helped me
debug various issues I was having with trying to make background_psql
work reliably for this test on 15. This was truly painful, but I never
would have gotten through it without my IPC::Run support group.
I'll probably add more robust comments to the test next week in
preparation for writing a detailed commit message for the fix
explaining the scenario.
- Melanie
Attachments: download all (2 files, .tar.gz)
0001-vacuum-hang-repro.patch (text/x-patch, 8.5K) download
From 45e668172b296daa7db08c9c1a41f3813b7dd593 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 14 May 2024 10:35:43 -0400
Subject: [PATCH] vacuum hang repro
---
src/test/recovery/t/099_vacuum_hang.pl | 246 +++++++++++++++++++++++++
1 file changed, 246 insertions(+)
create mode 100644 src/test/recovery/t/099_vacuum_hang.pl
diff --git a/src/test/recovery/t/099_vacuum_hang.pl b/src/test/recovery/t/099_vacuum_hang.pl
new file mode 100644
index 00000000000..aac943ca2ea
--- /dev/null
+++ b/src/test/recovery/t/099_vacuum_hang.pl
@@ -0,0 +1,246 @@
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Test that connections to a hot standby are correctly canceled when a
+# recovery conflict is detected Also, test that statistics in
+# pg_stat_database_conflicts are populated correctly
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use IPC::Run qw(pump);
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+my $tablespace1 = "test_vacuum_hang_tblspc";
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+allow_in_place_tablespaces = on
+hot_standby_feedback = on
+autovacuum = off
+]);
+$node_primary->start;
+
+my $backup_name = 'my_backup';
+
+$node_primary->safe_psql('postgres',
+ qq[CREATE TABLESPACE $tablespace1 LOCATION '']);
+
+$node_primary->backup($backup_name);
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+$node_replica->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+my $orig_conninfo = $node_primary->connstr();
+
+# test schema / data
+my $table1 = "test_vacuum_hang_table";
+my $index1 = "test_vacuum_hang_index";
+my $col1 = "col1";
+
+my $psql_timeout = IPC::Run::timer(10);
+
+# Long-running Primary Session A
+my %psql_primaryA = ('stdin' => '', 'stdout' => '');
+$psql_primaryA{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryA{stdin},
+ \$psql_primaryA{stdout},
+ $psql_timeout);
+$psql_primaryA{stdout} = '';
+
+# Long-running Primary Session B
+my %psql_primaryB = ('stdin' => '', 'stdout' => '');
+$psql_primaryB{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryB{stdin},
+ \$psql_primaryB{stdout},
+ $psql_timeout);
+$psql_primaryB{stdout} = '';
+
+# Long-running Primary Session C
+my %psql_primaryC = ('stdin' => '', 'stdout' => '');
+$psql_primaryC{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryC{stdin},
+ \$psql_primaryC{stdout},
+ $psql_timeout);
+$psql_primaryC{stdout} = '';
+
+# Long-running Replica Session A
+my %psql_replicaA = ('stdin' => '', 'stdout' => '');
+$psql_replicaA{run} =
+ $node_replica->background_psql($test_db, \$psql_replicaA{stdin},
+ \$psql_replicaA{stdout},
+ $psql_timeout);
+$psql_replicaA{stdout} = '';
+
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(${col1} int) with (autovacuum_enabled=false);
+ CREATE INDEX ${index1} on ${table1}(${col1});
+ INSERT INTO $table1 VALUES(1);
+ UPDATE $table1 SET $col1 = 100 WHERE $col1 = 1;
+]);
+
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+my $walreceiver_pid = $node_replica->safe_psql($test_db, qq[
+ select pid from pg_stat_activity where backend_type = 'walreceiver';]);
+
+# Set primary_conninfo to something invalid on the replica and reload the config.
+# This will prevent the standby from reconnecting once the connection is terminated.
+# Then terminate the wal receiver. When a new WAL receiver process starts up,
+# it will have to use the primary_conninfo to connect to the primary and be
+# unable to do so.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ SELECT pg_terminate_backend($walreceiver_pid)]);
+
+# Make sure the wal receiver is no longer listed
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_activity where pid = $walreceiver_pid);] , 'f');
+
+# DECLARE and use a cursor on standby, causing buffer with the only block of
+# the relation to be pinned on the standby
+my $replica_cursor1 = "test_vacuum_hang_cursor_replica1";
+$psql_replicaA{stdin} .= qq[
+ BEGIN;
+ DECLARE $replica_cursor1 CURSOR FOR SELECT * FROM $table1;
+ FETCH FORWARD FROM $replica_cursor1;
+ ];
+
+# FETCH FORWARD should have returned a 100
+ok(pump_until($psql_replicaA{run}, $psql_timeout,
+ \$psql_replicaA{stdout}, qr/^100$/m), "got first value from replica cursor");
+
+# Now we have two dead tuple versions. The xmax of the older tuple will be
+# older than OldestXmin and the xmax of the newer one will cause us to update
+# GlobalVisState, as its xmax will be between maybe_needed and
+# definitely_needed. We process it first during pruning.
+$psql_primaryA{stdin} .= qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET $col1 = 100 WHERE $col1 = 99;
+ SELECT 'after_update';
+ ];
+
+ok(pump_until($psql_primaryA{run}, $psql_timeout,
+ \$psql_primaryA{stdout}, qr/^after_update$/m), "SELECT output from primary session A");
+
+# This cursor is going to keep the ALTER INDEX SET TABLESPACE from acquiring an
+# exclusive lock on the index.
+my $primary_cursor1 = "test_vacuum_hang_cursor_primary1";
+$psql_primaryB{stdin} .= qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT * FROM $table1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ];
+
+ok(pump_until($psql_primaryB{run}, $psql_timeout,
+ \$psql_primaryB{stdout}, qr/^100$/m), "got first value from primary cursor");
+
+# This ALTER INDEX SET TABLESPACE will keep vacuum from proceeding after
+# initially calling vacuum_set_xid_limits(), it won't be able to get a lock on
+# the index.
+$psql_primaryC{stdin} .= qq[
+ BEGIN;
+ ALTER INDEX $index1 SET TABLESPACE $tablespace1;
+ SELECT 'after_alter_index';
+ ];
+
+$psql_primaryC{run}->pump_nb();
+
+# Make sure the ALTER INDEX is waiting for an exclusive lock on the index and
+# is unable to get it.
+$node_primary->poll_query_until($test_db, qq[
+ select granted from pg_locks
+ where mode = 'AccessExclusiveLock' and relation = '$index1'::regclass;], 'f');
+
+# Start the vacuum freeze. It will have to to get a lock on the index.
+$psql_primaryA{stdin} .= qq[
+ VACUUM (FREEZE) $table1;
+ \\echo VACUUM
+ ];
+
+$psql_primaryA{run}->pump_nb();
+
+# After committing the cursor, the ALTER INDEX should move forward
+$psql_primaryB{stdin} .= qq[
+ COMMIT;
+ \\echo commit
+ ];
+
+ok(pump_until($psql_primaryB{run}, $psql_timeout,
+ \$psql_primaryB{stdout}, qr/^commit$/m), "CURSOR committing");
+
+ok(pump_until($psql_primaryC{run}, $psql_timeout,
+ \$psql_primaryC{stdout}, qr/^after_alter_index$/m), "ALTER INDEX completed");
+
+# Allow the connection to re-establish
+# The new connection should cause a wal sender with an older oldest xmin to
+# push back the horizon on the primary.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Commit the alter index so that vacuum can move forward
+$psql_primaryC{stdin} .= qq[
+ COMMIT;
+ \\echo postcommit
+ ];
+
+ok(pump_until($psql_primaryC{run}, $psql_timeout,
+ \$psql_primaryC{stdout}, qr/^postcommit$/m), "FINISH ALTER INDEX. vacuum is free");
+
+$psql_primaryC{run}->pump_nb();
+
+$psql_primaryA{run}->pump_nb();
+
+pump $psql_primaryA{run} until ($psql_primaryA{stdout} =~ /VACUUM/ || $psql_timeout->is_expired);
+
+ok(!$psql_timeout->is_expired);
+
+# commit the original transaction on the replica so it can catch up
+$psql_replicaA{stdin} .= qq[
+ COMMIT;
+ ];
+
+$psql_replicaA{run}->pump_nb();
+
+$primary_lsn = $node_primary->lsn('flush');
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "insert into $table1 values (1);");
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Shut down psqls
+$psql_primaryA{stdin} .= "\\q\n";
+$psql_primaryA{run}->finish;
+
+$psql_primaryB{stdin} .= "\\q\n";
+$psql_primaryB{run}->finish;
+
+$psql_primaryC{stdin} .= "\\q\n";
+$psql_primaryC{run}->finish;
+
+$psql_replicaA{stdin} .= "\\q\n";
+$psql_replicaA{run}->finish;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
fix_hang_15.patch (text/x-patch, 4.9K) download
From eb5ed39ee00430755c45e4ed55ba1cde4c6fd79f Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 14 May 2024 10:35:14 -0400
Subject: [PATCH v1] fix
---
src/backend/access/heap/pruneheap.c | 36 +++++++++++++++++++---------
src/backend/access/heap/vacuumlazy.c | 2 +-
src/include/access/heapam.h | 1 +
3 files changed, 27 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 9f43bbe25f5..65d409e8b12 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -36,6 +36,12 @@ typedef struct
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
+ /*
+ * Used only by vacuum. The oldest xmin at the beginning of vacuuming the
+ * relation.
+ */
+ TransactionId oldest_xmin;
+
/*
* Thresholds set by TransactionIdLimitedForOldSnapshots() if they have
* been computed (done on demand, and only if
@@ -206,8 +212,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int ndeleted,
nnewlpdead;
- ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
- limited_ts, &nnewlpdead, NULL);
+ ndeleted = heap_page_prune(relation, buffer, vistest, InvalidTransactionId,
+ limited_xmin, limited_ts, &nnewlpdead, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -248,11 +254,14 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* also need to account for a reduction in the length of the line pointer
* array following array truncation by us.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum). old_snap_xmin / old_snap_ts need to
- * either have been set by TransactionIdLimitedForOldSnapshots, or
- * InvalidTransactionId/0 respectively.
+ * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD (see
+ * heap_prune_satisfies_vacuum and HeapTupleSatisfiesVacuum).
+ *
+ * oldest_xmin is only set to a valid TransactionId by vacuum. It is the oldest
+ * removable TransactionId at the beginning of vacuuming the relation.
+ *
+ * old_snap_xmin / old_snap_ts need to either have been set by
+ * TransactionIdLimitedForOldSnapshots, or InvalidTransactionId/0 respectively.
*
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
@@ -265,6 +274,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int
heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
+ TransactionId oldest_xmin,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
int *nnewlpdead,
@@ -292,6 +302,7 @@ heap_page_prune(Relation relation, Buffer buffer,
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
prstate.vistest = vistest;
+ prstate.oldest_xmin = oldest_xmin;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
@@ -520,11 +531,14 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
/*
- * First check if GlobalVisTestIsRemovableXid() is sufficient to find the
- * row dead. If not, and old_snapshot_threshold is enabled, try to use the
- * lowered horizon.
+ * First check if oldest_xmin or GlobalVisTestIsRemovableXid() is
+ * sufficient to find the row dead. If not, and old_snapshot_threshold is
+ * enabled, try to use the lowered horizon.
*/
- if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
+ if (TransactionIdIsValid(prstate->oldest_xmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ res = HEAPTUPLE_DEAD;
+ else if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
res = HEAPTUPLE_DEAD;
else if (OldSnapshotThresholdActive())
{
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2e61e2f20b4..a027458cf69 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1585,7 +1585,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest, vacrel->OldestXmin,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index abf62d9df79..e39fa14c1b2 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -186,6 +186,7 @@ struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern int heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
+ TransactionId oldest_xmin,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
int *nnewlpdead,
--
2.34.1
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-17 02:12 Bowen Shi <zxwsbg12138@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 62+ messages in thread
From: Bowen Shi @ 2024-05-17 02:12 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi
On Fri, May 17, 2024 at 12:38 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2024-05-16 15:31:20 +0800, Bowen Shi wrote:
> > Yes, I have preserved the coredump file of vacuum and several other
> related
> > processes. It has been at least 10 minutes stuck in the retry loop since
> I
> > dumped this core file.
>
> Could you print out *vacrel?
>
> I'm wondering if there was index processing, due to the number of tuples.
> And
> if so, what type of indexes. There'd need to be something that could lead
> to
> new snapshots being acquired...
>
(gdb) p * vacrel
$2 = {rel = 0x2403028, indrels = 0x234e550, nindexes = 12, bstrategy =
0x2368e28, pvs = 0x0,
aggressive = false, skipwithvm = true, consider_bypass_optimization =
false,
do_index_vacuuming = true, do_index_cleanup = true, do_rel_truncate =
true, cutoffs = {
relfrozenxid = 927, relminmxid = 1, OldestXmin = 1106, OldestMxact = 1,
FreezeLimit = 4244968402, MultiXactCutoff = 4289967297},
vistest = 0xe693f0 <GlobalVisDataRels>, NewRelfrozenXid = 1106,
NewRelminMxid = 1,
skippedallvis = false, dbname = 0x234e7f0 "db1", relnamespace = 0x234eb38
"public",
relname = 0x234eb48 "t1", indname = 0x0, blkno = 349565,
offnum = 10, phase = VACUUM_ERRCB_PHASE_SCAN_HEAP, verbose = false,
dead_items = 0x7fa22f48d040, rel_pages = 1302478, scanned_pages = 349566,
removed_pages = 0,
frozen_pages = 592, lpdead_item_pages = 349565, missed_dead_pages = 0,
nonempty_pages = 349495, new_rel_tuples = 0, new_live_tuples = 0,
indstats = 0x234e6e0,
num_index_scans = 1, tuples_deleted = 11184540, tuples_frozen = 1540,
lpdead_items = 11184540,
live_tuples = 1540, recently_dead_tuples = 0, missed_dead_tuples = 0}
(gdb) p * vacrel->bstrategy
$3 = {btype = BAS_VACUUM, nbuffers = 32, current = 30, buffers = 0x2368e34}
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-17 03:35 Bowen Shi <zxwsbg12138@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-17 03:35 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andres Freund <andres@anarazel.de>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On Fri, May 17, 2024 at 12:49 AM Peter Geoghegan <pg@bowt.ie> wrote:
> On Thu, May 16, 2024 at 12:38 PM Andres Freund <andres@anarazel.de> wrote:
> > I'm wondering if there was index processing, due to the number of
> tuples. And
> > if so, what type of indexes. There'd need to be something that could
> lead to
> > new snapshots being acquired...
>
> Did you ever see this theory of mine, about B-Tree page deletion +
> recycling? See:
>
>
> https://www.postgresql.org/message-id/flat/CAH2-Wz%3DzLcnZO8MqPXQLqOLY%3DCAwQhdvs5Ncg6qMb5nMAam0EA%4...
>
> (And related nearby emails from me.)
>
> It looked very much like index vacuuming was involved in some way when
> I actually had the opportunity to use gdb against an affected
> production instance that ran into the problem.
>
In my case, the call stack is following:
#0 0x000000000050d50e in heap_page_prune (relation=relation@entry=0x2403028,
buffer=buffer@entry=67521, vistest=<optimized out>,
old_snap_xmin=old_snap_xmin@entry=0,
old_snap_ts=old_snap_ts@entry=0,
nnewlpdead=nnewlpdead@entry=0x7fff8be98ecc,
off_loc=off_loc@entry=0x234e3cc)
#1 0x0000000000510678 in lazy_scan_prune (vacrel=vacrel@entry=0x234e348,
buf=buf@entry=67521,
blkno=blkno@entry=349565, page=page@entry=0x7fa25b728000 "'",
prunestate=prunestate@entry=0x7fff8be9a0d0)
#2 0x0000000000511a70 in lazy_scan_heap (vacrel=0x234e348)
#3 heap_vacuum_rel (rel=0x2403028, params=0x2358064, bstrategy=<optimized
out>)
#4 0x00000000006767e7 in table_relation_vacuum (bstrategy=0x2368e28,
params=0x2358064,
rel=0x2403028)
#5 vacuum_rel (relid=18930, relation=<optimized out>,
params=params@entry=0x2358064,
bstrategy=bstrategy@entry=0x2368e28)
#6 0x0000000000677be0 in vacuum (relations=0x23652f8,
relations@entry=0x2363310,
params=params@entry=0x2358064, bstrategy=bstrategy@entry=0x2368e28,
vac_context=vac_context@entry=0x23651a0, isTopLevel=isTopLevel@entry
=true)
#7 0x0000000000778080 in autovacuum_do_vac_analyze (bstrategy=0x2368e28,
tab=0x2358060)
#8 do_autovacuum ()
#9 0x0000000000778510 in AutoVacWorkerMain (argv=0x0, argc=0)
#10 0x00000000007785eb in StartAutoVacWorker ()
#11 0x000000000077efe1 in StartAutovacuumWorker ()
#12 process_pm_pmsignal ()
#13 ServerLoop ()
#14 0x0000000000780328 in PostmasterMain (argc=argc@entry=3, argv=argv@entry
=0x22583a0)
#15 0x00000000004bc368 in main (argc=3, argv=0x22583a0)
(gdb) p * MyProc
$4 = {links = {prev = 0x0, next = 0x0}, procgloballist = 0x7fa34153ecb8,
sem = 0x7fa237311138,
waitStatus = PROC_WAIT_STATUS_OK, procLatch = {is_set = 1, maybe_sleeping
= 0,
is_shared = true, owner_pid = 2303}, xid = 0, xmin = 1079, lxid =
237689, pid = 2303,
pgxactoff = 13, pgprocno = 2050, backendId = 13, databaseId = 16425,
roleId = 0,
tempNamespaceId = 0, isBackgroundWorker = false, recoveryConflictPending
= false,
lwWaiting = 0 '\000', lwWaitMode = 0 '\000', lwWaitLink = {next = 0, prev
= 0}, cvWaitLink = {
next = 0, prev = 0}, waitLock = 0x0, waitProcLock = 0x0, waitLockMode =
0, heldLocks = 0,
waitStart = {value = 0}, delayChkptFlags = 0, statusFlags = 3 '\003',
waitLSN = 0,
syncRepState = 0, syncRepLinks = {prev = 0x0, next = 0x0}, myProcLocks =
{{head = {
prev = 0x7fa3416f7518, next = 0x7fa3416f7518}}, {head = {prev =
0x7fa33e401d60,
next = 0x7fa33e401d60}}, {head = {prev = 0x7fa33e453db0, next =
0x7fa33e453db0}}, {
head = {prev = 0x7fa3416f7548, next = 0x7fa3416f7548}}, {head = {prev
= 0x7fa33ea10060,
next = 0x7fa33e4f7220}}, {head = {prev = 0x7fa33e548cd0, next =
0x7fa33e548cd0}}, {
head = {prev = 0x7fa33eab22b0, next = 0x7fa33eab2350}}, {head = {prev
= 0x7fa33e5f9020,
next = 0x7fa33eb11960}}, {head = {prev = 0x7fa3416f7598, next =
0x7fa3416f7598}}, {
head = {prev = 0x7fa33eba74f0, next = 0x7fa33eba74f0}}, {head = {prev
= 0x7fa3416f75b8,
next = 0x7fa3416f75b8}}, {head = {prev = 0x7fa3416f75c8, next =
0x7fa3416f75c8}}, {
head = {prev = 0x7fa33eca8c60, next = 0x7fa33eca89e0}}, {head = {prev
= 0x7fa3416f75e8,
next = 0x7fa3416f75e8}}, {head = {prev = 0x7fa3416f75f8, next =
0x7fa3416f75f8}}, {
head = {prev = 0x7fa33e884de0, next = 0x7fa33e884de0}}}, subxidStatus
= {count = 0 '\000',
overflowed = false}, subxids = {xids = {0 <repeats 64 times>}},
procArrayGroupMember = false, procArrayGroupNext = {value = 2147483647},
procArrayGroupMemberXid = 0, wait_event_info = 0, clogGroupMember =
false, clogGroupNext = {
value = 2147483647}, clogGroupMemberXid = 0, clogGroupMemberXidStatus =
0,
clogGroupMemberPage = -1, clogGroupMemberLsn = 0, fpInfoLock = {tranche =
81, state = {
value = 536870912}, waiters = {head = 2147483647, tail =
2147483647}}, fpLockBits = 0,
fpRelId = {2840, 2662, 2659, 3379, 2841, 2840, 2662, 27770, 26889, 26008,
24246, 23365, 2659,
1249, 2690, 53019}, fpVXIDLock = true, fpLocalTransactionId = 237689,
lockGroupLeader = 0x0,
lockGroupMembers = {head = {prev = 0x7fa3416f77b0, next =
0x7fa3416f77b0}}, lockGroupLink = {
prev = 0x0, next = 0x0}}
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-17 04:28 Peter Geoghegan <pg@bowt.ie>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Peter Geoghegan @ 2024-05-17 04:28 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, May 16, 2024 at 11:36 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
> From the current situation, I feel that the scenario I encountered is different from yours because it doesn't seem to be the first page of the vacuum scanning.
Really? I think it looks like the same issue.
What I saw was that the heap page that lazy_scan_prune locked up on
was the first page scanned *after* the first round of index vacuuming
-- *not* the first heap page scanned overall.
I also saw "num_index_scans == 1", "tuples_deleted == lpdead_items",
and "scanned_pages == lpdead_item_pages" when I looked at the locked
up Postgres 15 production instance (or close to it). I can't tell if
your dead_items is shown as empty, too, but if it is then that
indicates that the heap page that your lazy_scan_prune trace relates
to is the first page scanned after a round of index vacuuming (the
first round).
If you can show more about dead_items, then it will either confirm or
disprove my theory about it being the same issue.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-18 22:23 Noah Misch <noah@leadboat.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-05-18 22:23 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, May 16, 2024 at 07:57:27PM -0400, Melanie Plageman wrote:
> On Thu, May 9, 2024 at 5:56 PM Melanie Plageman <melanieplageman@gmail.com> wrote:
> > I can repro the hang on 14 and 15 with the following:
> I'll probably add more robust comments to the test next week in
> preparation for writing a detailed commit message for the fix
> explaining the scenario.
Are there obstacles to fixing the hang by back-patching 1ccc1e05ae instead of
this? We'll need to get confident about 1ccc1e05ae before v17, and that
sounds potentially easier than getting confident about both 1ccc1e05ae and
this other approach. Regarding getting confident about 1ccc1e05ae, I think I
follow the upthread arguments that it does operate correctly. As a cross
check, I looked at each mention of oldestxmin in vacuumlazy.c and vacuum.c.
Does the following heap_vacuum_rel() comment need an update?
/*
* Get cutoffs that determine which deleted tuples are considered DEAD,
* not just RECENTLY_DEAD, and which XIDs/MXIDs to freeze. Then determine
* the extent of the blocks that we'll scan in lazy_scan_heap. It has to
* happen in this order to ensure that the OldestXmin cutoff field works
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
* Next acquire vistest, a related cutoff that's used in pruning. We
* expect vistest will always make heap_page_prune_and_freeze() remove any
* deleted tuple whose xmax is < OldestXmin.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-20 15:58 Melanie Plageman <melanieplageman@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-05-20 15:58 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, May 18, 2024 at 6:23 PM Noah Misch <noah@leadboat.com> wrote:
>
> Are there obstacles to fixing the hang by back-patching 1ccc1e05ae instead of
> this? We'll need to get confident about 1ccc1e05ae before v17, and that
> sounds potentially easier than getting confident about both 1ccc1e05ae and
> this other approach.
I haven't tried back-patching 1ccc1e05ae yet, but I don't understand
why we would want to use stable back branches to get comfortable with
an approach committed to an unreleased version of Postgres.
The small fix proposed in this thread is extremely minimal and
straightforward. It seems much less risky as a backpatch.
> Regarding getting confident about 1ccc1e05ae, I think I
> follow the upthread arguments that it does operate correctly. As a cross
> check, I looked at each mention of oldestxmin in vacuumlazy.c and vacuum.c.
> Does the following heap_vacuum_rel() comment need an update?
>
> /*
> * Get cutoffs that determine which deleted tuples are considered DEAD,
> * not just RECENTLY_DEAD, and which XIDs/MXIDs to freeze. Then determine
> * the extent of the blocks that we'll scan in lazy_scan_heap. It has to
> * happen in this order to ensure that the OldestXmin cutoff field works
> * as an upper bound on the XIDs stored in the pages we'll actually scan
> * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
> *
> * Next acquire vistest, a related cutoff that's used in pruning. We
> * expect vistest will always make heap_page_prune_and_freeze() remove any
> * deleted tuple whose xmax is < OldestXmin.
Which part are you thinking needs an update? The part about vistest
always making heap_page_prune_and_freeze() remove any deleted tuples
whose xmax < OldestXmin?
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-20 20:48 Noah Misch <noah@leadboat.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-05-20 20:48 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, May 20, 2024 at 11:58:23AM -0400, Melanie Plageman wrote:
> On Sat, May 18, 2024 at 6:23 PM Noah Misch <noah@leadboat.com> wrote:
> > Are there obstacles to fixing the hang by back-patching 1ccc1e05ae instead of
> > this? We'll need to get confident about 1ccc1e05ae before v17, and that
> > sounds potentially easier than getting confident about both 1ccc1e05ae and
> > this other approach.
>
> I haven't tried back-patching 1ccc1e05ae yet, but I don't understand
> why we would want to use stable back branches to get comfortable with
> an approach committed to an unreleased version of Postgres.
I wouldn't say we want to use stable back branches to get comfortable with an
approach. I wanted to say that it's less work to be confident about one new
way of doing things than two new ways of doing things.
> The small fix proposed in this thread is extremely minimal and
> straightforward. It seems much less risky as a backpatch.
Here's how I model the current and proposed code:
1. v15 VACUUM removes tuples that are HEAPTUPLE_DEAD according to VisTest.
OldestXmin doesn't cause tuple removal, but there's a hang when OldestXmin
rules DEAD after VisTest ruled RECENTLY_DEAD.
2. With 1ccc1e05ae, v17 VACUUM still removes tuples that are HEAPTUPLE_DEAD
according to VisTest only. OldestXmin doesn't come into play.
3. fix_hang_15.patch would make v15 VACUUM remove tuples that are
HEAPTUPLE_DEAD according to _either_ VisTest or OldestXmin.
Since (3) is the only list entry that removes tuples that VisTest ruled
RECENTLY_DEAD, I find it higher risk. For all three, the core task of
certifying the behavior is confirming that its outcome is sound when VisTest
and OldestXmin disagree. How do you model it?
> > Regarding getting confident about 1ccc1e05ae, I think I
> > follow the upthread arguments that it does operate correctly. As a cross
> > check, I looked at each mention of oldestxmin in vacuumlazy.c and vacuum.c.
> > Does the following heap_vacuum_rel() comment need an update?
> >
> > /*
> > * Get cutoffs that determine which deleted tuples are considered DEAD,
> > * not just RECENTLY_DEAD, and which XIDs/MXIDs to freeze. Then determine
Since 1ccc1e05ae, the cutoffs (values acquired in vacuum_get_cutoffs()) don't
decide DEAD.
> > * the extent of the blocks that we'll scan in lazy_scan_heap. It has to
Still accurate.
> > * happen in this order to ensure that the OldestXmin cutoff field works
> > * as an upper bound on the XIDs stored in the pages we'll actually scan
That sentence's problem predates 1ccc1e05ae. (OldestXmin is a lower bound on
XIDs in pages we don't scan, not an upper bound on XIDs in pages we scan. To
be pedantic, snapshots don't affect this bound, so the oldest running XID is
also a lower bound.)
> > * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
Still accurate.
> > *
> > * Next acquire vistest, a related cutoff that's used in pruning. We
Still accurate.
> > * expect vistest will always make heap_page_prune_and_freeze() remove any
> > * deleted tuple whose xmax is < OldestXmin.
Since 1ccc1e05ae, that sentence does not apply.
> Which part are you thinking needs an update? The part about vistest
> always making heap_page_prune_and_freeze() remove any deleted tuples
> whose xmax < OldestXmin?
That and one other sentence marked above.
(Long-term, I favor removal of the cutoffs->OldestXmin field. I'm not aware
of any calculation for which it remains the optimal value.)
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-22 16:57 Melanie Plageman <melanieplageman@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 4 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-05-22 16:57 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, May 20, 2024 at 4:48 PM Noah Misch <noah@leadboat.com> wrote:
>
> On Mon, May 20, 2024 at 11:58:23AM -0400, Melanie Plageman wrote:
> > On Sat, May 18, 2024 at 6:23 PM Noah Misch <noah@leadboat.com> wrote:
> > > Are there obstacles to fixing the hang by back-patching 1ccc1e05ae instead of
> > > this? We'll need to get confident about 1ccc1e05ae before v17, and that
> > > sounds potentially easier than getting confident about both 1ccc1e05ae and
> > > this other approach.
> >
> > I haven't tried back-patching 1ccc1e05ae yet, but I don't understand
> > why we would want to use stable back branches to get comfortable with
> > an approach committed to an unreleased version of Postgres.
>
> I wouldn't say we want to use stable back branches to get comfortable with an
> approach. I wanted to say that it's less work to be confident about one new
> way of doing things than two new ways of doing things.
I think I understand your point better now.
> > The small fix proposed in this thread is extremely minimal and
> > straightforward. It seems much less risky as a backpatch.
>
> Here's how I model the current and proposed code:
>
> 1. v15 VACUUM removes tuples that are HEAPTUPLE_DEAD according to VisTest.
> OldestXmin doesn't cause tuple removal, but there's a hang when OldestXmin
> rules DEAD after VisTest ruled RECENTLY_DEAD.
>
> 2. With 1ccc1e05ae, v17 VACUUM still removes tuples that are HEAPTUPLE_DEAD
> according to VisTest only. OldestXmin doesn't come into play.
>
> 3. fix_hang_15.patch would make v15 VACUUM remove tuples that are
> HEAPTUPLE_DEAD according to _either_ VisTest or OldestXmin.
>
> Since (3) is the only list entry that removes tuples that VisTest ruled
> RECENTLY_DEAD, I find it higher risk. For all three, the core task of
> certifying the behavior is confirming that its outcome is sound when VisTest
> and OldestXmin disagree. How do you model it?
Okay, I see your point. In 1 and 2, tuples that would have been
considered HEAPTUPLE_DEAD at the beginning of vacuum but are
considered HEAPTUPLE_RECENTLY_DEAD when pruning them are not removed.
In 3, tuples that would have been considered HEAPTUPLE_DEAD at the
beginning of vacuum are always removed, regardless of whether or not
they would be considered HEAPTUPLE_RECENTLY_DEAD when pruning them.
One option is to add the logic in fix_hang_15.patch to master as well
(always remove tuples older than OldestXmin). This addresses your
concern about gaining confidence in a single solution.
However, I can see how removing more tuples could be concerning. In
the case that the horizon moves backwards because of a standby
reconnecting, I think the worst case is that removing that tuple
causes a recovery conflict on the standby (depending on the value of
max_standby_streaming_delay et al).
I'll experiment with applying 1ccc1e05ae to 14 and 15 and see how it goes.
> > > Regarding getting confident about 1ccc1e05ae, I think I
> > > follow the upthread arguments that it does operate correctly. As a cross
> > > check, I looked at each mention of oldestxmin in vacuumlazy.c and vacuum.c.
> > > Does the following heap_vacuum_rel() comment need an update?
> > >
> > > /*
> > > * Get cutoffs that determine which deleted tuples are considered DEAD,
> > > * not just RECENTLY_DEAD, and which XIDs/MXIDs to freeze. Then determine
>
> Since 1ccc1e05ae, the cutoffs (values acquired in vacuum_get_cutoffs()) don't
> decide DEAD.
Thinking about it more, this is misleading either way. The values
acquired in vacuum_get_cutoffs() don't determine which tuples are
removed because they are considered HEAPTUPLE_DEAD. If they are
considered HEAPTUPLE_RECENTLY_DEAD by VisTest and HEAPTUPLE_DEAD when
compared to OldestXmin, there will be a hang.
> > > * the extent of the blocks that we'll scan in lazy_scan_heap. It has to
>
> Still accurate.
I thought about this more, and I actually don't understand what "Then
determine the extent of the blocks that we'll scan in lazy_scan_heap"
means. I assume it refers to RelationGetNumberOfBlocks() which is
called after vacuum_get_cutoffs(). But, if so, I have no idea how that
relates to the following sentence which says that it "has to happen in
this order to ensure that the OldestXmin cutoff field works as an
upper bound"...
> > > * happen in this order to ensure that the OldestXmin cutoff field works
> > > * as an upper bound on the XIDs stored in the pages we'll actually scan
>
> That sentence's problem predates 1ccc1e05ae. (OldestXmin is a lower bound on
> XIDs in pages we don't scan, not an upper bound on XIDs in pages we scan. To
> be pedantic, snapshots don't affect this bound, so the oldest running XID is
> also a lower bound.)
Hmm. I'm not sure I agree with your interpretation here. Or perhaps I
don't understand it. We initialize NewRelfrozenXid with OldestXmin and
that is, in turn, used to initialize pagefrz.NoFreezePageRelfrozenXid.
HeapPageFreeze->NoFreezePageRelfrozenXid is updated with any older xid
found in an unfrozen tuple on the pages we _do_ scan. That makes
OldestXmin an upper bound on unfrozen xids on the pages we do scan.
I don't think we can say anything about the XIDs of tuples on pages we
don't scan. We skip pages if they are all visible. We don't know that
tuples on those pages are newer than OldestXmin. That's why we can't
update relfrozenxid when we skip pages that are not all frozen.
And even if we skip scanning pages because they are all frozen, their
xids will be older than OldestXmin, so it wouldn't be a lower bound on
those either.
I don't understand "snapshots don't affect this bound, so the oldest
running XID is also a lower bound".
> > > * (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
>
> Still accurate.
Agree
> > > *
> > > * Next acquire vistest, a related cutoff that's used in pruning. We
>
> Still accurate.
Agree.
> > > * expect vistest will always make heap_page_prune_and_freeze() remove any
> > > * deleted tuple whose xmax is < OldestXmin.
>
> Since 1ccc1e05ae, that sentence does not apply.
That's true. I wonder how we should modify it. Some options:
1) change it from "always" to "usually"
2) remove the sentence because it is not worth mentioning OldestXmin
in relation to removing tuples
3) explain the situation with GlobalVisTest moving the horizon
backwards to before OldestXmin
4) change it to say vistest will always cause pruning to remove
deleted tuples visible as deleted to everyone
I also think we should change this part:
* lazy_scan_prune must never
* become confused about whether a tuple should be frozen or removed. (In
* the future we might want to teach lazy_scan_prune to recompute vistest
* from time to time, to increase the number of dead tuples it can prune
* away.)
lazy_scan_prune() can't really become confused about whether a tuple
should be frozen or removed. It may choose to freeze a tuple that was
ruled RECENTLY_DEAD by vistest and not removed. But that is okay. I
don't think there is any confusion.
The second part I'm not sure about either. Recomputing vistest in
lazy_scan_prune() would not cause more tuples to be removed. Doing so
in heap_page_prune_and_freeze() for reasons other than the current
logic in GlobalVisTestShouldUpdate() would have this effect. But, I
can't tell if that is even what this comment is suggesting.
> (Long-term, I favor removal of the cutoffs->OldestXmin field. I'm not aware
> of any calculation for which it remains the optimal value.)
I'm trying to think if there is any complication with using
GlobalVisState for freezing given that it could be updated during the
course of vacuuming the relation. If we used GlobalVisState when
determining what to freeze and maybe_needed moves backwards, perhaps
we could end up incorrectly updating relfrozenxid?
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-22 23:59 Noah Misch <noah@leadboat.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
3 siblings, 0 replies; 62+ messages in thread
From: Noah Misch @ 2024-05-22 23:59 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, May 22, 2024 at 12:57:26PM -0400, Melanie Plageman wrote:
> On Mon, May 20, 2024 at 4:48 PM Noah Misch <noah@leadboat.com> wrote:
> > On Mon, May 20, 2024 at 11:58:23AM -0400, Melanie Plageman wrote:
> > > On Sat, May 18, 2024 at 6:23 PM Noah Misch <noah@leadboat.com> wrote:
> > > > I looked at each mention of oldestxmin in vacuumlazy.c and vacuum.c.
> > > > Does the following heap_vacuum_rel() comment need an update?
> > > >
> > > > /*
> > > > * Get cutoffs that determine which deleted tuples are considered DEAD,
> > > > * not just RECENTLY_DEAD, and which XIDs/MXIDs to freeze. Then determine
> >
> > Since 1ccc1e05ae, the cutoffs (values acquired in vacuum_get_cutoffs()) don't
> > decide DEAD.
>
> Thinking about it more, this is misleading either way. The values
> acquired in vacuum_get_cutoffs() don't determine which tuples are
> removed because they are considered HEAPTUPLE_DEAD. If they are
> considered HEAPTUPLE_RECENTLY_DEAD by VisTest and HEAPTUPLE_DEAD when
> compared to OldestXmin, there will be a hang.
Agreed.
> > > > * the extent of the blocks that we'll scan in lazy_scan_heap. It has to
> >
> > Still accurate.
>
> I thought about this more, and I actually don't understand what "Then
> determine the extent of the blocks that we'll scan in lazy_scan_heap"
> means. I assume it refers to RelationGetNumberOfBlocks() which is
> called after vacuum_get_cutoffs().
I think it does refer to that.
> But, if so, I have no idea how that
> relates to the following sentence which says that it "has to happen in
> this order to ensure that the OldestXmin cutoff field works as an
> upper bound"...
When the comment says "pages we'll actually scan", it means "pages that
existed at the start of VACUUM" or equivalently "pages of the rel except those
pages extended after this point in time". I bet that phrase of the comment
predates the other ways VACUUM can now opt not to visit specific pages.
The lines where order is important:
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
Suppose the first line sets vacrel->cutoffs.OldestXmin=1234 and the second
line sets orig_rel_pages=10. Suppose, by the end of the VACUUM, the rel size
has grown to 15 pages. The last 5 pages can contain only XIDs >= 1234. We
can assume that without scanning those pages because of the order of those two
lines. If we swapped the order of those two lines, XID=1233 could extend the
rel and commit between the two lines, without any cutoff reflecting that.
> I don't think we can say anything about the XIDs of tuples on pages we
> don't scan. We skip pages if they are all visible. We don't know that
> tuples on those pages are newer than OldestXmin. That's why we can't
> update relfrozenxid when we skip pages that are not all frozen.
Agreed. We can infer bounds only for newly-extended pages, not for pages not
scanned for some other reason.
> And even if we skip scanning pages because they are all frozen, their
> xids will be older than OldestXmin, so it wouldn't be a lower bound on
> those either.
>
> I don't understand "snapshots don't affect this bound, so the oldest
> running XID is also a lower bound".
Assume OldestXmin=1M and OldestXid=2M, because there's a read-only transaction
with an old snapshot. If something extends the rel after this code runs,
newly-extended pages won't add any transaction older than xid=2M.
> > > > * expect vistest will always make heap_page_prune_and_freeze() remove any
> > > > * deleted tuple whose xmax is < OldestXmin.
> >
> > Since 1ccc1e05ae, that sentence does not apply.
>
> That's true. I wonder how we should modify it. Some options:
>
> 1) change it from "always" to "usually"
> 2) remove the sentence because it is not worth mentioning OldestXmin
> in relation to removing tuples
> 3) explain the situation with GlobalVisTest moving the horizon
> backwards to before OldestXmin
> 4) change it to say vistest will always cause pruning to remove
> deleted tuples visible as deleted to everyone
Any of (2),(3), or (4) works for me.
> I also think we should change this part:
>
> * lazy_scan_prune must never
> * become confused about whether a tuple should be frozen or removed. (In
> * the future we might want to teach lazy_scan_prune to recompute vistest
> * from time to time, to increase the number of dead tuples it can prune
> * away.)
>
> lazy_scan_prune() can't really become confused about whether a tuple
> should be frozen or removed. It may choose to freeze a tuple that was
> ruled RECENTLY_DEAD by vistest and not removed. But that is okay. I
> don't think there is any confusion.
Agreed.
> The second part I'm not sure about either. Recomputing vistest in
> lazy_scan_prune() would not cause more tuples to be removed. Doing so
> in heap_page_prune_and_freeze() for reasons other than the current
> logic in GlobalVisTestShouldUpdate() would have this effect. But, I
> can't tell if that is even what this comment is suggesting.
I read it as the following general idea. If other backends update their xmin
values and/or end their transactions while a VACUUM is running, VACUUM could
opt to prune more as that news arrives. VACUUM could check for such news
before starting on each page or whatever.
> > (Long-term, I favor removal of the cutoffs->OldestXmin field. I'm not aware
> > of any calculation for which it remains the optimal value.)
For avoidance of doubt, I don't think this thread should make such a change.
> I'm trying to think if there is any complication with using
> GlobalVisState for freezing given that it could be updated during the
> course of vacuuming the relation. If we used GlobalVisState when
> determining what to freeze and maybe_needed moves backwards, perhaps
> we could end up incorrectly updating relfrozenxid?
I think it suffices to calculate relfrozenxid=min(OldestXid at VACUUM start,
all XIDs actually present in pages). OldestXid might still insert a tuple on
any page of the rel. OldestXid might not even start until after VACUUM
finishes. Hence, we can't set any relfrozenxid lower than OldestXid. But a
transaction with an old snapshot and no XID doesn't constrain relfrozenxid.
If that transaction does assign an XID, it will assign a value >= OldestXid.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-28 06:31 Bowen Shi <zxwsbg12138@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-28 06:31 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andres Freund <andres@anarazel.de>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On Fri, May 17, 2024 at 12:29 PM Peter Geoghegan <pg@bowt.ie> wrote:
> On Thu, May 16, 2024 at 11:36 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
> > From the current situation, I feel that the scenario I encountered is
> different from yours because it doesn't seem to be the first page of the
> vacuum scanning.
>
> Really? I think it looks like the same issue.
>
> What I saw was that the heap page that lazy_scan_prune locked up on
> was the first page scanned *after* the first round of index vacuuming
> -- *not* the first heap page scanned overall.
>
> I misunderstood earlier.
> I also saw "num_index_scans == 1", "tuples_deleted == lpdead_items",
> and "scanned_pages == lpdead_item_pages" when I looked at the locked
> up Postgres 15 production instance (or close to it). I can't tell if
> your dead_items is shown as empty, too, but if it is then that
> indicates that the heap page that your lazy_scan_prune trace relates
> to is the first page scanned after a round of index vacuuming (the
> first round).
>
> If you can show more about dead_items, then it will either confirm or
> disprove my theory about it being the same issue.
>
(gdb) p *vacrel->dead_items
$2 = {max_items = 11184809, num_items = 0, items = 0x7fa22f48d048}
(gdb) p *vacrel->dead_items->items
$3 = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 1}
The num_items is zero.
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-28 06:38 Peter Geoghegan <pg@bowt.ie>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 0 replies; 62+ messages in thread
From: Peter Geoghegan @ 2024-05-28 06:38 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Melanie Plageman <melanieplageman@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Noah Misch <noah@leadboat.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, May 27, 2024 at 11:32 PM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>> If you can show more about dead_items, then it will either confirm or
>> disprove my theory about it being the same issue.
>
>
> (gdb) p *vacrel->dead_items
> $2 = {max_items = 11184809, num_items = 0, items = 0x7fa22f48d048}
> (gdb) p *vacrel->dead_items->items
> $3 = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 1}
>
> The num_items is zero.
That is what I suspected. And so the issue you see looks exactly the
same as the one that I saw when debugging a production instance.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-28 08:29 Bowen Shi <zxwsbg12138@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
3 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-28 08:29 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
I only had time to check the fix_hang_15.patch until today.
On Thu, May 23, 2024 at 12:57 AM Melanie Plageman <melanieplageman@gmail.com>
wrote:
> On Mon, May 20, 2024 at 4:48 PM Noah Misch <noah@leadboat.com> wrote:
> >
> > On Mon, May 20, 2024 at 11:58:23AM -0400, Melanie Plageman wrote:
> > > On Sat, May 18, 2024 at 6:23 PM Noah Misch <noah@leadboat.com> wrote:
> > > > Are there obstacles to fixing the hang by back-patching 1ccc1e05ae
> instead of
> > > > this? We'll need to get confident about 1ccc1e05ae before v17, and
> that
> > > > sounds potentially easier than getting confident about both
> 1ccc1e05ae and
> > > > this other approach.
> > >
> > > I haven't tried back-patching 1ccc1e05ae yet, but I don't understand
> > > why we would want to use stable back branches to get comfortable with
> > > an approach committed to an unreleased version of Postgres.
> >
> > I wouldn't say we want to use stable back branches to get comfortable
> with an
> > approach. I wanted to say that it's less work to be confident about one
> new
> > way of doing things than two new ways of doing things.
>
> I think I understand your point better now.
>
> > > The small fix proposed in this thread is extremely minimal and
> > > straightforward. It seems much less risky as a backpatch.
> >
> > Here's how I model the current and proposed code:
> >
> > 1. v15 VACUUM removes tuples that are HEAPTUPLE_DEAD according to
> VisTest.
> > OldestXmin doesn't cause tuple removal, but there's a hang when
> OldestXmin
> > rules DEAD after VisTest ruled RECENTLY_DEAD.
> >
> > 2. With 1ccc1e05ae, v17 VACUUM still removes tuples that are
> HEAPTUPLE_DEAD
> > according to VisTest only. OldestXmin doesn't come into play.
> >
> > 3. fix_hang_15.patch would make v15 VACUUM remove tuples that are
> > HEAPTUPLE_DEAD according to _either_ VisTest or OldestXmin.
> >
> > Since (3) is the only list entry that removes tuples that VisTest ruled
> > RECENTLY_DEAD, I find it higher risk. For all three, the core task of
> > certifying the behavior is confirming that its outcome is sound when
> VisTest
> > and OldestXmin disagree. How do you model it?
>
> Okay, I see your point. In 1 and 2, tuples that would have been
> considered HEAPTUPLE_DEAD at the beginning of vacuum but are
> considered HEAPTUPLE_RECENTLY_DEAD when pruning them are not removed.
> In 3, tuples that would have been considered HEAPTUPLE_DEAD at the
> beginning of vacuum are always removed, regardless of whether or not
> they would be considered HEAPTUPLE_RECENTLY_DEAD when pruning them.
>
> One option is to add the logic in fix_hang_15.patch to master as well
> (always remove tuples older than OldestXmin). This addresses your
> concern about gaining confidence in a single solution.
>
> However, I can see how removing more tuples could be concerning. In
> the case that the horizon moves backwards because of a standby
> reconnecting, I think the worst case is that removing that tuple
> causes a recovery conflict on the standby (depending on the value of
> max_standby_streaming_delay et al).
>
What would happen if we simply skipped the current page when we found the
vacuum process had entered the infinite loop (use a counter)?
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-05-28 09:03 Bowen Shi <zxwsbg12138@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
3 siblings, 1 reply; 62+ messages in thread
From: Bowen Shi @ 2024-05-28 09:03 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On Thu, May 23, 2024 at 12:57 AM Melanie Plageman <melanieplageman@gmail.com>
wrote:
> One option is to add the logic in fix_hang_15.patch to master as well
> (always remove tuples older than OldestXmin). This addresses your
> concern about gaining confidence in a single solution.
>
> However, I can see how removing more tuples could be concerning. In
> the case that the horizon moves backwards because of a standby
> reconnecting, I think the worst case is that removing that tuple
> causes a recovery conflict on the standby (depending on the value of
> max_standby_streaming_delay et al).
>
I'm confused about this part. The comment above OldestXmin says:
/*
* OldestXmin is the Xid below which tuples deleted by any xact (that
* committed) should be considered DEAD, not just RECENTLY_DEAD.
*/
With the fix_hang_15 patch, why is there a risk here when we use Oldestxmin
to judge whether a tuple could be moved?
I think the keypoint is: OldestXmin and VisTest, which one is more accurate
when we judge to remove the tuple.
--
Regards
Bowen Shi
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-06-17 21:29 Melanie Plageman <melanieplageman@gmail.com>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 0 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-06-17 21:29 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, May 28, 2024 at 4:29 AM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>
> Hi,
>
> I only had time to check the fix_hang_15.patch until today.
Did it end up working for you?
> What would happen if we simply skipped the current page when we found the vacuum process had entered the infinite loop (use a counter)?
Skipping the page would mean we can't advance relfrozenxid for the
table on this vacuum. Since we have ways to fix it, it seems better to
do that.
- Melanie
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-06-18 22:39 Melanie Plageman <melanieplageman@gmail.com>
parent: Bowen Shi <zxwsbg12138@gmail.com>
0 siblings, 0 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-06-18 22:39 UTC (permalink / raw)
To: Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, May 28, 2024 at 5:03 AM Bowen Shi <zxwsbg12138@gmail.com> wrote:
>
> On Thu, May 23, 2024 at 12:57 AM Melanie Plageman <melanieplageman@gmail.com> wrote:
>>
>> One option is to add the logic in fix_hang_15.patch to master as well
>> (always remove tuples older than OldestXmin). This addresses your
>> concern about gaining confidence in a single solution.
>>
>> However, I can see how removing more tuples could be concerning. In
>> the case that the horizon moves backwards because of a standby
>> reconnecting, I think the worst case is that removing that tuple
>> causes a recovery conflict on the standby (depending on the value of
>> max_standby_streaming_delay et al).
>
>
> I'm confused about this part. The comment above OldestXmin says:
> /*
> * OldestXmin is the Xid below which tuples deleted by any xact (that
> * committed) should be considered DEAD, not just RECENTLY_DEAD.
> */
> With the fix_hang_15 patch, why is there a risk here when we use Oldestxmin to judge whether a tuple could be moved?
The risk is not so much about whether or not it is okay to remove the
tuple. At least for the case of a lagging standby reconnecting during
vacuum and moving maybe_needed backwards, there is no real danger in
removing tuples considered non-removable when compared to the new
value of maybe_needed. The standby will not replay the tuple removal
until the tuple is removable. If the horizon doesn't move forward on
the standby after some time, a recovery conflict will occur --
canceling the culprit holding back the horizon and allowing the vacuum
to proceed.
I have not, however, investigated the second way maybe_needed may go
backwards cited by Matthias in [1] to see if it is similarly okay to
remove tuples older than OldestXmin but newer than maybe_needed in
this case.
I think Noah's concern about using OldestXmin is more that it is a
different behavior than what we have on master. Picking one solution
and standardizing on it would reduce maintenance complexity across all
branches.
You make a good point about the comment though. If OldestXmin is only
used during vacuum for freezing tuples and determining full page
visibility status, then we should update the comment above it in
VacuumCutoffs and remove mention of HEAPTUPLE_DEAD.
I'll propose both this comment update and the one Noah suggested to
heap_vacuum_rel() against master in a separate thread. Then we can
backport the comments (as relevant) when we fix the back branches.
> I think the keypoint is: OldestXmin and VisTest, which one is more accurate when we judge to remove the tuple.
GlobalVisState will have a potentially more up-to-date horizon value
than OldestXmin when determining whether or not to remove the tuple.
But I wouldn't call it incorrect to remove a tuple older than
OldestXmin but younger than maybe_needed.
- Melanie
[1] https://www.postgresql.org/message-id/CAEze2WjMTh4KS0%3DQEQB-Jq%2BtDLPR%2B0%2BzVBMfVwSPK5A%3DWZa95Q%...
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-06-18 22:51 Melanie Plageman <melanieplageman@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
3 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-06-18 22:51 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, May 22, 2024 at 12:57 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
>
> On Mon, May 20, 2024 at 4:48 PM Noah Misch <noah@leadboat.com> wrote:
> >
> > On Mon, May 20, 2024 at 11:58:23AM -0400, Melanie Plageman wrote:
> > > On Sat, May 18, 2024 at 6:23 PM Noah Misch <noah@leadboat.com> wrote:
> > > > Are there obstacles to fixing the hang by back-patching 1ccc1e05ae instead of
> > > > this? We'll need to get confident about 1ccc1e05ae before v17, and that
> > > > sounds potentially easier than getting confident about both 1ccc1e05ae and
> > > > this other approach.
> > >
> > > I haven't tried back-patching 1ccc1e05ae yet, but I don't understand
> > > why we would want to use stable back branches to get comfortable with
> > > an approach committed to an unreleased version of Postgres.
> >
> > I wouldn't say we want to use stable back branches to get comfortable with an
> > approach. I wanted to say that it's less work to be confident about one new
> > way of doing things than two new ways of doing things.
>
> I think I understand your point better now.
>
> > > The small fix proposed in this thread is extremely minimal and
> > > straightforward. It seems much less risky as a backpatch.
> >
> > Here's how I model the current and proposed code:
> >
> > 1. v15 VACUUM removes tuples that are HEAPTUPLE_DEAD according to VisTest.
> > OldestXmin doesn't cause tuple removal, but there's a hang when OldestXmin
> > rules DEAD after VisTest ruled RECENTLY_DEAD.
> >
> > 2. With 1ccc1e05ae, v17 VACUUM still removes tuples that are HEAPTUPLE_DEAD
> > according to VisTest only. OldestXmin doesn't come into play.
> >
> > 3. fix_hang_15.patch would make v15 VACUUM remove tuples that are
> > HEAPTUPLE_DEAD according to _either_ VisTest or OldestXmin.
> >
> > Since (3) is the only list entry that removes tuples that VisTest ruled
> > RECENTLY_DEAD, I find it higher risk. For all three, the core task of
> > certifying the behavior is confirming that its outcome is sound when VisTest
> > and OldestXmin disagree. How do you model it?
>
> Okay, I see your point. In 1 and 2, tuples that would have been
> considered HEAPTUPLE_DEAD at the beginning of vacuum but are
> considered HEAPTUPLE_RECENTLY_DEAD when pruning them are not removed.
> In 3, tuples that would have been considered HEAPTUPLE_DEAD at the
> beginning of vacuum are always removed, regardless of whether or not
> they would be considered HEAPTUPLE_RECENTLY_DEAD when pruning them.
>
> One option is to add the logic in fix_hang_15.patch to master as well
> (always remove tuples older than OldestXmin). This addresses your
> concern about gaining confidence in a single solution.
>
> However, I can see how removing more tuples could be concerning. In
> the case that the horizon moves backwards because of a standby
> reconnecting, I think the worst case is that removing that tuple
> causes a recovery conflict on the standby (depending on the value of
> max_standby_streaming_delay et al).
>
> I'll experiment with applying 1ccc1e05ae to 14 and 15 and see how it goes.
I ended up manually backporting the logic from 1ccc1e05ae as opposed
to cherry-picking because it relied on a struct introduced in
4e9fc3a9762065. Attached is a patch set with this backport against
REL_15_STABLE. The first patch is an updated repro (now even more
minimal) with copious additional comments. I am not proposing we add
this as an ongoing test. It won't be stable. It is purely for
illustration.
The fix's commit message still needs editing and citations.
My repro no longer works against REL_14_STABLE, though I was able to
backport the fix there. I'll investigate that.
There are some comments that should be updated around OldestXmin if we
go with this approach. I think they should be updated first in master
and then backported. So, I'll start a thread on those.
Finally, upthread there is discussion of how we could end up doing a
catalog lookup after vacuum_get_cutoffs() and before the tuple
visibility check on 16. Assuming this is true, we would want to
backport the fix to 16 as well. I could use some help getting a repro
(using btree index deletion for example) of the infinite loop on 16.
- Melanie
Attachments: download all (2 files, .tar.gz)
v1-0002-Fix-infinite-loop-in-vacuum-caused-by-visibility-.patch (text/x-patch, 13.7K) download
From 088e68b7afc7c8b601ac0cef24d962b13a71ec3b Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 18 Jun 2024 16:06:48 -0400
Subject: [PATCH v1 2/2] Fix infinite loop in vacuum caused by visibility
mismatch
lazy_scan_prune() may loop infinitely if a tuple deemed non-removable by
heap_prune_satisfies_vacuum() in heap_page_prune() is later deemed
HEAPTUPLE_DEAD in lazy_scan_prune() by HeapTupleSatisfiesVacuum().
Both determine what transaction ID a tuple would be dead_after using
HeapTupleSatisfiesVacuumHorizon() and then compare that transaction ID
to a horizon. HeapTupleSatisfiesVacuum() uses OldestXmin from
VacuumCutoffs as that horizon and heap_prune_satisfies_vacuum() uses
GlobalVisTestIsRemovableXid() -- which checks the GlobalVisState that
was passed in to heap_page_prune().
OldestXmin is determined at the start of vacuuming the relation, but
GlobalVisState's maybe_needed value can change if a tuple's xmax falls
within GlobalVisState's fuzzy window and vacuum has had to take a new
snapshot since it started vacuuming the relation.
In rare cases, maybe_needed may move backward -- for example if a lagging
standby which was not connected at the start of the vacuum later
connects.
When this happened, a tuple with an xmax older than OldestXmin but newer
than the new value of maybe_needed after GlobalVisState was updated
could cause lazy_scan_prune() to loop infinitely.
By only computing the tuple's HTSV_Result once, in heap_page_prune(),
and saving it to be passed back to lazy_scan_prune(), we can avoid the
problem altogether. This also happens to be cheaper, since
HeapTupleSatisfiesVacuumHorizon() accesses the clog.
As a consequence, OldestXmin is now only used for determining whether or
not to freeze tuples and whether or not the whole page is visible -- not
for determining whether or not tuples are removable.
---
src/backend/access/heap/pruneheap.c | 47 ++++++++++++++++------------
src/backend/access/heap/vacuumlazy.c | 46 ++++++++++-----------------
src/include/access/heapam.h | 15 +++++++++
3 files changed, 58 insertions(+), 50 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 9f43bbe25f5..801cba51bc6 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -65,16 +65,6 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
-
- /*
- * Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
- *
- * Same indexing as ->marked.
- */
- int8 htsv[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
@@ -83,7 +73,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate);
+ PruneState *prstate, int8 *htsv);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -205,9 +195,10 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
{
int ndeleted,
nnewlpdead;
+ int8 htsv[MaxHeapTuplesPerPage + 1];
ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
- limited_ts, &nnewlpdead, NULL);
+ limited_ts, htsv, &nnewlpdead, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -254,6 +245,18 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* either have been set by TransactionIdLimitedForOldSnapshots, or
* InvalidTransactionId/0 respectively.
*
+ * htsv is an in/out parameter filled in by heap_page_prune() with tuple
+ * visibility, computed once for each tuple for correctness and efficiency
+ * reasons. Some callers want access to htsv for later decisions and metrics.
+ * htsv is of type int8[], instead of HTSV_Result[], so we can use -1 to
+ * indicate no visibility has been computed, e.g. for LP_DEAD items. It can be
+ * passed in uninitialized because all spots will be set either to a valid
+ * HTSV_Result value or -1.
+ *
+ * It is the caller's responsibility to ensure that htsv is
+ * MaxHeapTuplesPerPage + 1 in size. By making it MaxHeapTuplesPerPage + 1, we
+ * can avoid subtracting 1 everywhere even though FirstOffsetNumber is 1.
+ *
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
*
@@ -267,6 +270,7 @@ heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
+ int8 htsv[MaxHeapTuplesPerPage + 1],
int *nnewlpdead,
OffsetNumber *off_loc)
{
@@ -332,7 +336,7 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Nothing to do if slot doesn't contain a tuple */
if (!ItemIdIsNormal(itemid))
{
- prstate.htsv[offnum] = -1;
+ htsv[offnum] = -1;
continue;
}
@@ -348,7 +352,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
+ htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
buffer);
}
@@ -373,7 +377,7 @@ heap_page_prune(Relation relation, Buffer buffer,
continue;
/* Process this item or chain of items */
- ndeleted += heap_prune_chain(buffer, offnum, &prstate);
+ ndeleted += heap_prune_chain(buffer, offnum, &prstate, htsv);
}
/* Clear the offset information once we have processed the given page. */
@@ -561,6 +565,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
* Prune specified line pointer or a HOT chain originating at line pointer.
*
+ * Tuple visibility information is provided in htsv.
+ *
* If the item is an index-referenced tuple (i.e. not a heap-only tuple),
* the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
* chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
@@ -588,7 +594,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
* Returns the number of tuples (to be) deleted from the page.
*/
static int
-heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
+heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate,
+ int8 *htsv)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -609,7 +616,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
if (ItemIdIsNormal(rootlp))
{
- Assert(prstate->htsv[rootoffnum] != -1);
+ Assert(htsv[rootoffnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
if (HeapTupleHeaderIsHeapOnly(htup))
@@ -632,7 +639,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* either here or while following a chain below. Whichever path
* gets there first will mark the tuple unused.
*/
- if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
+ if (htsv[rootoffnum] == HEAPTUPLE_DEAD &&
!HeapTupleHeaderIsHotUpdated(htup))
{
heap_prune_record_unused(prstate, rootoffnum);
@@ -700,7 +707,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
Assert(ItemIdIsNormal(lp));
- Assert(prstate->htsv[offnum] != -1);
+ Assert(htsv[offnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, lp);
/*
@@ -720,7 +727,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
tupdead = recent_dead = false;
- switch ((HTSV_Result) prstate->htsv[offnum])
+ switch ((HTSV_Result) htsv[offnum])
{
case HEAPTUPLE_DEAD:
tupdead = true;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2e61e2f20b4..8ba0da063db 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1526,12 +1526,14 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* of complexity just so we could deal with tuples that were DEAD to VACUUM,
* but nevertheless were left with storage after pruning.
*
- * The approach we take now is to restart pruning when the race condition is
- * detected. This allows heap_page_prune() to prune the tuples inserted by
- * the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * As of minor release blah, we circumvent this problem altogether by reusing
+ * the result of heap_page_prune()'s visibility check. Without the second call
+ * to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be
+ * no disagreement. We'll just handle such tuples as if they had become fully
+ * dead right after this operation completes instead of in the middle of it.
+ * Note that any tuple that becomes dead after the call to heap_page_prune()
+ * can't need to be frozen, because it was visible to another session when
+ * vacuum started.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1544,8 +1546,6 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- HeapTupleData tuple;
- HTSV_Result res;
int tuples_deleted,
lpdead_items,
live_tuples,
@@ -1556,6 +1556,7 @@ lazy_scan_prune(LVRelState *vacrel,
MultiXactId NewRelminMxid;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+ int8 htsv[MaxHeapTuplesPerPage + 1];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1566,8 +1567,6 @@ lazy_scan_prune(LVRelState *vacrel,
*/
maxoff = PageGetMaxOffsetNumber(page);
-retry:
-
/* Initialize (or reset) page-level state */
NewRelfrozenXid = vacrel->NewRelfrozenXid;
NewRelminMxid = vacrel->NewRelminMxid;
@@ -1586,7 +1585,7 @@ retry:
* that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
- InvalidTransactionId, 0, &nnewlpdead,
+ InvalidTransactionId, 0, htsv, &nnewlpdead,
&vacrel->offnum);
/*
@@ -1604,6 +1603,7 @@ retry:
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
+ HeapTupleHeader htup;
bool tuple_totally_frozen;
/*
@@ -1649,21 +1649,7 @@ retry:
Assert(ItemIdIsNormal(itemid));
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
- tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
- tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationGetRelid(rel);
-
- /*
- * DEAD tuples are almost always pruned into LP_DEAD line pointers by
- * heap_page_prune(), but it's possible that the tuple state changed
- * since heap_page_prune() looked. Handle that here by restarting.
- * (See comments at the top of function for a full explanation.)
- */
- res = HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf);
-
- if (unlikely(res == HEAPTUPLE_DEAD))
- goto retry;
+ htup = (HeapTupleHeader) PageGetItem(page, itemid);
/*
* The criteria for counting a tuple as live in this block need to
@@ -1684,7 +1670,7 @@ retry:
* (Cases where we bypass index vacuuming will violate this optimistic
* assumption, but the overall impact of that should be negligible.)
*/
- switch (res)
+ switch (htsv_get_valid_status(htsv[offnum]))
{
case HEAPTUPLE_LIVE:
@@ -1706,7 +1692,7 @@ retry:
{
TransactionId xmin;
- if (!HeapTupleHeaderXminCommitted(tuple.t_data))
+ if (!HeapTupleHeaderXminCommitted(htup))
{
prunestate->all_visible = false;
break;
@@ -1716,7 +1702,7 @@ retry:
* The inserter definitely committed. But is it old enough
* that everyone sees it as committed?
*/
- xmin = HeapTupleHeaderGetXmin(tuple.t_data);
+ xmin = HeapTupleHeaderGetXmin(htup);
if (!TransactionIdPrecedes(xmin, vacrel->OldestXmin))
{
prunestate->all_visible = false;
@@ -1772,7 +1758,7 @@ retry:
* now.
*/
prunestate->hastup = true; /* page makes rel truncation unsafe */
- if (heap_prepare_freeze_tuple(tuple.t_data,
+ if (heap_prepare_freeze_tuple(htup,
vacrel->relfrozenxid,
vacrel->relminmxid,
vacrel->FreezeLimit,
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index abf62d9df79..0b349427982 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -188,6 +188,7 @@ extern int heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
+ int8 htsv[MaxHeapTuplesPerPage + 1],
int *nnewlpdead,
OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
@@ -230,4 +231,18 @@ extern bool ResolveCminCmaxDuringDecoding(struct HTAB *tuplecid_data,
extern void HeapCheckForSerializableConflictOut(bool valid, Relation relation, HeapTuple tuple,
Buffer buffer, Snapshot snapshot);
+/*
+ * Pruning calculates tuple visibility once and saves the results in an array
+ * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * guard against examining visibility status array members which have not yet
+ * been computed.
+ */
+static inline HTSV_Result
+htsv_get_valid_status(int status)
+{
+ Assert(status >= HEAPTUPLE_DEAD &&
+ status <= HEAPTUPLE_DELETE_IN_PROGRESS);
+ return (HTSV_Result) status;
+}
+
#endif /* HEAPAM_H */
--
2.34.1
v1-0001-Repro-for-vacuum-infinite-loop.patch (text/x-patch, 11.2K) download
From 7aab24e8bff9f9d7f0ab4247bc4cb727a33aa2a5 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 18 Jun 2024 16:09:00 -0400
Subject: [PATCH v1 1/2] Repro for vacuum infinite loop
This repro is not stable enough to be added as a test to the regression
suite. It is for demonstration purposes on the thread.
---
src/test/recovery/t/099_vacuum_hang.pl | 286 +++++++++++++++++++++++++
1 file changed, 286 insertions(+)
create mode 100644 src/test/recovery/t/099_vacuum_hang.pl
diff --git a/src/test/recovery/t/099_vacuum_hang.pl b/src/test/recovery/t/099_vacuum_hang.pl
new file mode 100644
index 00000000000..55272c39b23
--- /dev/null
+++ b/src/test/recovery/t/099_vacuum_hang.pl
@@ -0,0 +1,286 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use IPC::Run qw(pump);
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+my $tablespace1 = "test_vacuum_hang_tblspc";
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+allow_in_place_tablespaces = on
+hot_standby_feedback = on
+log_recovery_conflict_waits = true
+log_statement='all'
+log_connections=true
+log_lock_waits = true
+autovacuum = off
+]);
+$node_primary->start;
+
+my $backup_name = 'my_backup';
+
+# Make a tablespace for use with our later ALTER INDEX ... SET TABLESPACE
+$node_primary->safe_psql('postgres',
+ qq[CREATE TABLESPACE $tablespace1 LOCATION '']);
+
+$node_primary->backup($backup_name);
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+$node_replica->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+my $orig_conninfo = $node_primary->connstr();
+
+# test schema / data
+my $table1 = "test_vacuum_hang_table";
+my $index1 = "test_vacuum_hang_index";
+my $col1 = "col1";
+
+my $psql_timeout = IPC::Run::timer(10);
+
+# Long-running Primary Session A
+my %psql_primaryA = ('stdin' => '', 'stdout' => '');
+$psql_primaryA{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryA{stdin},
+ \$psql_primaryA{stdout},
+ $psql_timeout);
+$psql_primaryA{stdout} = '';
+$psql_primaryA{stdin} = "set application_name=A;\n";
+
+# Long-running Primary Session B
+my %psql_primaryB = ('stdin' => '', 'stdout' => '');
+$psql_primaryB{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryB{stdin},
+ \$psql_primaryB{stdout},
+ $psql_timeout);
+$psql_primaryB{stdout} = '';
+$psql_primaryB{stdin} = "set application_name=B;\n";
+
+# Long-running Primary Session C
+my %psql_primaryC = ('stdin' => '', 'stdout' => '');
+$psql_primaryC{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryC{stdin},
+ \$psql_primaryC{stdout},
+ $psql_timeout);
+$psql_primaryC{stdout} = '';
+$psql_primaryC{stdin} = "set application_name=C;\n";
+
+# Long-running Replica Session A
+my %psql_replicaA = ('stdin' => '', 'stdout' => '');
+$psql_replicaA{run} =
+ $node_replica->background_psql($test_db, \$psql_replicaA{stdin},
+ \$psql_replicaA{stdout},
+ $psql_timeout);
+$psql_replicaA{stdout} = '';
+
+# Insert one tuple with value 1 which we can use to make sure the cursor has
+# successfully pinned and locked the buffer.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(${col1} int) with (autovacuum_enabled=false);
+ CREATE INDEX ${index1} on ${table1}(${col1});
+ INSERT INTO $table1 VALUES(1);
+]);
+
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+my $walreceiver_pid = $node_replica->safe_psql($test_db, qq[
+ select pid from pg_stat_activity where backend_type = 'walreceiver';]);
+
+# Set primary_conninfo to something invalid on the replica and reload the config.
+# This will prevent the standby from reconnecting once the connection is terminated.
+# Then terminate the wal receiver. When a new WAL receiver process starts up,
+# it will have to use the primary_conninfo to connect to the primary and be
+# unable to do so.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ SELECT pg_terminate_backend($walreceiver_pid)]);
+
+# Ensure the WAL receiver is no longer active on replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_activity where pid = $walreceiver_pid);] , 'f');
+
+# DECLARE and use a cursor on standby, causing the block of the relation to be
+# pinned and locked in a buffer on the replica. It is important that this is
+# after termination of the WAL receiver so that the primary does not know about
+# the cursor and it can't hold back the horizon on the primary.
+my $replica_cursor1 = "test_vacuum_hang_cursor_replica1";
+$psql_replicaA{stdin} .= qq[
+ BEGIN;
+ DECLARE $replica_cursor1 CURSOR FOR SELECT * FROM $table1;
+ FETCH FORWARD FROM $replica_cursor1;
+ ];
+
+# FETCH FORWARD should have returned a 1. That's how we know the cursor has a
+# pin and lock.
+ok(pump_until($psql_replicaA{run}, $psql_timeout,
+ \$psql_replicaA{stdout}, qr/^1$/m), "got first value from replica cursor");
+
+# Insert a new tuple and update it. Now there are two live tuples and one dead
+# tuple on the page on the primary. The connection to the replica was
+# terminated, so the replica will only have the single live tuple.
+$psql_primaryA{stdin} .= qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET $col1 = 100 WHERE $col1 = 99;
+ SELECT 'after_update';
+ ];
+
+# Make sure the UPDATE finished
+ok(pump_until($psql_primaryA{run}, $psql_timeout,
+ \$psql_primaryA{stdout}, qr/^after_update$/m), "SELECT output from primary session A");
+
+# Open a cursor on the primary whose lock will keep a subsequent ALTER INDEX
+# from executing. We want VACUUM to be able to start, calculate initial values
+# for OldestXmin and GlobalVisState and then pause before pruning our dead
+# tuple. To accomplish this, we will run an ALTER INDEX which starts before
+# VACUUM but cannot proceed (because of the cursor's lock on the index) until
+# after VACUUM gets initial values for OldestXmin and GlobalVisState. When the
+# cursor is closed, the ALTER INDEX will proceed and modify the index, causing
+# a relcache invalidation and forcing vacuum to rebuild the catalog snapshot
+# after the ALTER INDEX commits before pruning the dead tuple.
+#
+my $primary_cursor1 = "test_vacuum_hang_cursor_primary1";
+$psql_primaryB{stdin} .= qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT * FROM $table1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ];
+
+# There is one live tuple on the primary with value 100 and one live tuple with
+# value 1.
+ok(pump_until($psql_primaryB{run}, $psql_timeout,
+ \$psql_primaryB{stdout}, qr/^1$/m), "got first value from primary cursor");
+
+# the ALTER INDEX won't proceed until the cursor has been closed because it
+# needs an exclusive lock and the cursor has an access share lock.
+$psql_primaryC{stdin} .= qq[
+ BEGIN;
+ ALTER INDEX $index1 SET TABLESPACE $tablespace1;
+ SELECT 'after_alter_index';
+ ];
+
+$psql_primaryC{run}->pump_nb();
+
+# Wait until we know the access exclusive lock has been requested but not granted
+$node_primary->poll_query_until($test_db, qq[
+ select granted from pg_locks
+ where mode = 'AccessExclusiveLock' and relation = '$index1'::regclass;], 'f');
+
+# ALTER INDEX has a row exclusive lock on the index and share update exclusive
+# lock on the table, so VACUUM will have to wait while attempting to open the
+# indexes.
+#
+# Note that all transactions on the primary see the dead tuple on the primary
+# as committed, so before the new snapshot is taken and GlobalVisState is
+# updated, VACUUM's GlobalVisState->maybe_needed will be newer than the dead
+# tuple's xmax, so the tuple would have been considered removable when compared
+# to GlobalVisState.
+$psql_primaryA{stdin} .= qq[
+ VACUUM $table1;
+ \\echo VACUUM
+ ];
+
+# Try and make sure our vacuum command has reached the server before we commit
+# the cursor.
+$psql_primaryA{run}->pump_nb();
+
+# After committing the cursor, the ALTER INDEX should proceed. We don't commit
+# its outer transaction, however, so VACUUM will still be waiting.
+$psql_primaryB{stdin} .= qq[
+ COMMIT;
+ \\echo commit
+ ];
+
+ok(pump_until($psql_primaryB{run}, $psql_timeout,
+ \$psql_primaryB{stdout}, qr/^commit$/m), "CURSOR committing");
+
+ok(pump_until($psql_primaryC{run}, $psql_timeout,
+ \$psql_primaryC{stdout}, qr/^after_alter_index$/m), "ALTER INDEX ran");
+
+# Allow the WAL receiver connection to re-establish. VACUUM is still
+# waiting for the ALTER INDEX to commit.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the GlobalVisUpdate() though.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Commit the transaction surrounding the ALTER INDEX
+$psql_primaryC{stdin} .= qq[
+ COMMIT;
+ \\echo postcommit
+ ];
+
+# VACUUM will take a new snapshot since the index modification caused a
+# relcache invalidation. The new value of RecentXmin should reflect the
+# older horizon from the standby and it will differ with
+# ComputeXidHorizonsResultLastXmin, causing us to update
+# GlobalVisState->maybe_needed to an older value.
+ok(pump_until($psql_primaryC{run}, $psql_timeout,
+ \$psql_primaryC{stdout}, qr/^postcommit$/m), "FINISH ALTER INDEX");
+
+$psql_primaryC{run}->pump_nb();
+
+$psql_primaryA{run}->pump_nb();
+
+# VACUUM proceeds to pruning and does a visibility check on each tuple. It will
+# find our dead tuple non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is
+# after the new value of maybe_needed. Once vacuum finishes the first pruning
+# pass, it will, in lazy_scan_prune(), do another visibility check, this time
+# with HeapTupleSatisfiesVacuum() which compares dead_after to OldestXmin. It
+# will find the tuple HEAPTUPLE_DEAD since its xmax precedes OldestXmin. This
+# will cause the infinite loop.
+pump $psql_primaryA{run} until ($psql_primaryA{stdout} =~ /VACUUM/ || $psql_timeout->is_expired);
+
+ok(!$psql_timeout->is_expired);
+
+# Commit the original cursor transaction on the replica so it can catch up. it
+# will end up replaying the vacuum and not removing the tuple too.
+$psql_replicaA{stdin} .= qq[
+ COMMIT;
+ ];
+
+$psql_replicaA{run}->pump_nb();
+
+$primary_lsn = $node_primary->lsn('flush');
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "insert into $table1 values (1);");
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Shut down psqls
+$psql_primaryA{stdin} .= "\\q\n";
+$psql_primaryA{run}->finish;
+
+$psql_primaryB{stdin} .= "\\q\n";
+$psql_primaryB{run}->finish;
+
+$psql_primaryC{stdin} .= "\\q\n";
+$psql_primaryC{run}->finish;
+
+$psql_replicaA{stdin} .= "\\q\n";
+$psql_replicaA{run}->finish;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-06-20 15:49 Melanie Plageman <melanieplageman@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 2 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-06-20 15:49 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 18, 2024 at 6:51 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
>
> I ended up manually backporting the logic from 1ccc1e05ae as opposed
> to cherry-picking because it relied on a struct introduced in
> 4e9fc3a9762065. Attached is a patch set with this backport against
> REL_15_STABLE. The first patch is an updated repro (now even more
> minimal) with copious additional comments. I am not proposing we add
> this as an ongoing test. It won't be stable. It is purely for
> illustration.
> The fix's commit message still needs editing and citations.
>
> My repro no longer works against REL_14_STABLE, though I was able to
> backport the fix there. I'll investigate that.
I figured out why it wasn't repro-ing on 14 -- just a timing issue. I
threw in a sleep for now. There is also something I can do with
pg_stat_progress_vacuum's phase column in the repro, but that was
harder without poll_query_until() (added in 16).
> Finally, upthread there is discussion of how we could end up doing a
> catalog lookup after vacuum_get_cutoffs() and before the tuple
> visibility check on 16. Assuming this is true, we would want to
> backport the fix to 16 as well. I could use some help getting a repro
> (using btree index deletion for example) of the infinite loop on 16.
So, I ended up working on a new repro that works by forcing a round of
index vacuuming after the standby reconnects and before pruning a dead
tuple whose xmax is older than OldestXmin.
At the end of the round of index vacuuming, _bt_pendingfsm_finalize()
calls GetOldestNonRemovableTransactionId(), thereby updating the
backend's GlobalVisState and moving maybe_needed backwards.
Then vacuum's first pass will continue with pruning and find our later
inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
I make sure that the standby reconnects between vacuum_get_cutoffs()
(vacuum_set_xid_limits() on 14/15) and pruning because I have a cursor
on the page keeping VACUUM FREEZE from getting a cleanup lock.
See the repros for step-by-step explanations of how it works.
With this, I can repro the infinite loop on 14-16.
Backporting 1ccc1e05ae fixes 16 but, with the new repro, 14 and 15
error out with "cannot freeze committed xmax". I'm going to
investigate further why this is happening. It definitely makes me
wonder about the fix.
Attached is the backport and repros for 15 and 16. Note that because
of differences in background_psl, the perl test had to be different in
15 than 16, so you'll have to use the repro targeted at the correct
version
- Melanie
Attachments: download all (4 files, .tar.gz)
REL16_v1-0002-Fix-infinite-loop-in-vacuum-caused-by-visibility-.patch (text/x-patch, 14.0K) download
From b2aeed6d4b83b46e818423e674b182256843ccf4 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 18 Jun 2024 16:06:48 -0400
Subject: [PATCH v1 2/2] Fix infinite loop in vacuum caused by visibility
mismatch
lazy_scan_prune() may loop infinitely if a tuple deemed non-removable by
heap_prune_satisfies_vacuum() in heap_page_prune() is later deemed
HEAPTUPLE_DEAD in lazy_scan_prune() by HeapTupleSatisfiesVacuum().
Both determine what transaction ID a tuple would be dead_after using
HeapTupleSatisfiesVacuumHorizon() and then compare that transaction ID
to a horizon. HeapTupleSatisfiesVacuum() uses OldestXmin from
VacuumCutoffs as that horizon and heap_prune_satisfies_vacuum() uses
GlobalVisTestIsRemovableXid() -- which checks the GlobalVisState that
was passed in to heap_page_prune().
OldestXmin is determined at the start of vacuuming the relation, but
GlobalVisState's maybe_needed value can change if a tuple's xmax falls
within GlobalVisState's fuzzy window and vacuum has had to take a new
snapshot since it started vacuuming the relation.
In rare cases, a backend's GlobalVisState->maybe_needed may move
backward -- for example if a standby with an older horizon which was not
connected at the start of the the vacuum later connects.
When this happends, a tuple with an xmax older than OldestXmin but newer
than the new value of maybe_needed after GlobalVisState was updated
could cause lazy_scan_prune() to loop infinitely.
By only computing the tuple's HTSV_Result once, in heap_page_prune(),
and saving it to be passed back to lazy_scan_prune(), we can avoid the
problem altogether. This also happens to be cheaper, since
HeapTupleSatisfiesVacuumHorizon() accesses the clog.
As a consequence, OldestXmin is now only used for determining whether or
not to freeze tuples and whether or not the whole page is visible -- not
for determining whether or not tuples are removable.
---
src/backend/access/heap/pruneheap.c | 47 +++++++++++++++-----------
src/backend/access/heap/vacuumlazy.c | 50 ++++++++++------------------
src/include/access/heapam.h | 15 +++++++++
3 files changed, 59 insertions(+), 53 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 47b9e209154..404cdd13080 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -65,16 +65,6 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
-
- /*
- * Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
- *
- * Same indexing as ->marked.
- */
- int8 htsv[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
@@ -83,7 +73,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate);
+ PruneState *prstate, int8 *htsv);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -204,9 +194,10 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
{
int ndeleted,
nnewlpdead;
+ int8 htsv[MaxHeapTuplesPerPage + 1];
ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
- limited_ts, &nnewlpdead, NULL);
+ limited_ts, htsv, &nnewlpdead, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -253,6 +244,18 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* either have been set by TransactionIdLimitedForOldSnapshots, or
* InvalidTransactionId/0 respectively.
*
+ * htsv is an in/out parameter filled in by heap_page_prune() with tuple
+ * visibility, computed once for each tuple for correctness and efficiency
+ * reasons. Some callers want access to htsv for later decisions and metrics.
+ * htsv is of type int8[], instead of HTSV_Result[], so we can use -1 to
+ * indicate no visibility has been computed, e.g. for LP_DEAD items. It can be
+ * passed in uninitialized because all spots will be set either to a valid
+ * HTSV_Result value or -1.
+ *
+ * It is the caller's responsibility to ensure that htsv is
+ * MaxHeapTuplesPerPage + 1 in size. By making it MaxHeapTuplesPerPage + 1, we
+ * can avoid subtracting 1 everywhere even though FirstOffsetNumber is 1.
+ *
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
*
@@ -266,6 +269,7 @@ heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
+ int8 htsv[MaxHeapTuplesPerPage + 1],
int *nnewlpdead,
OffsetNumber *off_loc)
{
@@ -331,7 +335,7 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Nothing to do if slot doesn't contain a tuple */
if (!ItemIdIsNormal(itemid))
{
- prstate.htsv[offnum] = -1;
+ htsv[offnum] = -1;
continue;
}
@@ -347,7 +351,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
+ htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
buffer);
}
@@ -372,7 +376,7 @@ heap_page_prune(Relation relation, Buffer buffer,
continue;
/* Process this item or chain of items */
- ndeleted += heap_prune_chain(buffer, offnum, &prstate);
+ ndeleted += heap_prune_chain(buffer, offnum, &prstate, htsv);
}
/* Clear the offset information once we have processed the given page. */
@@ -561,6 +565,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
* Prune specified line pointer or a HOT chain originating at line pointer.
*
+ * Tuple visibility information is provided in htsv.
+ *
* If the item is an index-referenced tuple (i.e. not a heap-only tuple),
* the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
* chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
@@ -588,7 +594,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
* Returns the number of tuples (to be) deleted from the page.
*/
static int
-heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
+heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate,
+ int8 *htsv)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -609,7 +616,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
if (ItemIdIsNormal(rootlp))
{
- Assert(prstate->htsv[rootoffnum] != -1);
+ Assert(htsv[rootoffnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
if (HeapTupleHeaderIsHeapOnly(htup))
@@ -632,7 +639,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* either here or while following a chain below. Whichever path
* gets there first will mark the tuple unused.
*/
- if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
+ if (htsv[rootoffnum] == HEAPTUPLE_DEAD &&
!HeapTupleHeaderIsHotUpdated(htup))
{
heap_prune_record_unused(prstate, rootoffnum);
@@ -700,7 +707,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
Assert(ItemIdIsNormal(lp));
- Assert(prstate->htsv[offnum] != -1);
+ Assert(htsv[offnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, lp);
/*
@@ -720,7 +727,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
tupdead = recent_dead = false;
- switch ((HTSV_Result) prstate->htsv[offnum])
+ switch ((HTSV_Result) htsv[offnum])
{
case HEAPTUPLE_DEAD:
tupdead = true;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index c474b061146..2e2af40b5e0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1518,12 +1518,14 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* of complexity just so we could deal with tuples that were DEAD to VACUUM,
* but nevertheless were left with storage after pruning.
*
- * The approach we take now is to restart pruning when the race condition is
- * detected. This allows heap_page_prune() to prune the tuples inserted by
- * the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * As of minor release blah, we circumvent this problem altogether by reusing
+ * the result of heap_page_prune()'s visibility check. Without the second call
+ * to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be
+ * no disagreement. We'll just handle such tuples as if they had become fully
+ * dead right after this operation completes instead of in the middle of it.
+ * Note that any tuple that becomes dead after the call to heap_page_prune()
+ * can't need to be frozen, because it was visible to another session when
+ * vacuum started.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1536,8 +1538,6 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- HeapTupleData tuple;
- HTSV_Result res;
int tuples_deleted,
tuples_frozen,
lpdead_items,
@@ -1548,6 +1548,7 @@ lazy_scan_prune(LVRelState *vacrel,
int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
+ int8 htsv[MaxHeapTuplesPerPage + 1];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1558,8 +1559,6 @@ lazy_scan_prune(LVRelState *vacrel,
*/
maxoff = PageGetMaxOffsetNumber(page);
-retry:
-
/* Initialize (or reset) page-level state */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
@@ -1582,7 +1581,7 @@ retry:
* that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
- InvalidTransactionId, 0, &nnewlpdead,
+ InvalidTransactionId, 0, htsv, &nnewlpdead,
&vacrel->offnum);
/*
@@ -1599,6 +1598,7 @@ retry:
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
+ HeapTupleHeader htup;
bool totally_frozen;
/*
@@ -1641,22 +1641,7 @@ retry:
Assert(ItemIdIsNormal(itemid));
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
- tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
- tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationGetRelid(rel);
-
- /*
- * DEAD tuples are almost always pruned into LP_DEAD line pointers by
- * heap_page_prune(), but it's possible that the tuple state changed
- * since heap_page_prune() looked. Handle that here by restarting.
- * (See comments at the top of function for a full explanation.)
- */
- res = HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin,
- buf);
-
- if (unlikely(res == HEAPTUPLE_DEAD))
- goto retry;
+ htup = (HeapTupleHeader) PageGetItem(page, itemid);
/*
* The criteria for counting a tuple as live in this block need to
@@ -1677,7 +1662,7 @@ retry:
* (Cases where we bypass index vacuuming will violate this optimistic
* assumption, but the overall impact of that should be negligible.)
*/
- switch (res)
+ switch (htsv_get_valid_status(htsv[offnum]))
{
case HEAPTUPLE_LIVE:
@@ -1699,7 +1684,7 @@ retry:
{
TransactionId xmin;
- if (!HeapTupleHeaderXminCommitted(tuple.t_data))
+ if (!HeapTupleHeaderXminCommitted(htup))
{
prunestate->all_visible = false;
break;
@@ -1709,9 +1694,8 @@ retry:
* The inserter definitely committed. But is it old enough
* that everyone sees it as committed?
*/
- xmin = HeapTupleHeaderGetXmin(tuple.t_data);
- if (!TransactionIdPrecedes(xmin,
- vacrel->cutoffs.OldestXmin))
+ xmin = HeapTupleHeaderGetXmin(htup);
+ if (!TransactionIdPrecedes(xmin, vacrel->cutoffs.OldestXmin))
{
prunestate->all_visible = false;
break;
@@ -1763,7 +1747,7 @@ retry:
prunestate->hastup = true; /* page makes rel truncation unsafe */
/* Tuple with storage -- consider need to freeze */
- if (heap_prepare_freeze_tuple(tuple.t_data, &vacrel->cutoffs, &pagefrz,
+ if (heap_prepare_freeze_tuple(htup, &vacrel->cutoffs, &pagefrz,
&frozen[tuples_frozen], &totally_frozen))
{
/* Save prepared freeze plan for later */
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index faf50265191..c0d0592163b 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -288,6 +288,7 @@ extern int heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
+ int8 htsv[MaxHeapTuplesPerPage + 1],
int *nnewlpdead,
OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
@@ -329,4 +330,18 @@ extern bool ResolveCminCmaxDuringDecoding(struct HTAB *tuplecid_data,
extern void HeapCheckForSerializableConflictOut(bool visible, Relation relation, HeapTuple tuple,
Buffer buffer, Snapshot snapshot);
+/*
+ * Pruning calculates tuple visibility once and saves the results in an array
+ * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * guard against examining visibility status array members which have not yet
+ * been computed.
+ */
+static inline HTSV_Result
+htsv_get_valid_status(int status)
+{
+ Assert(status >= HEAPTUPLE_DEAD &&
+ status <= HEAPTUPLE_DELETE_IN_PROGRESS);
+ return (HTSV_Result) status;
+}
+
#endif /* HEAPAM_H */
--
2.34.1
REL15_v1-0002-Fix-infinite-loop-in-vacuum-caused-by-visibility-.patch (text/x-patch, 13.7K) download
From 3f4c145f1903da7a8cef3da03ca4fa110c4c0c0d Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 18 Jun 2024 16:06:48 -0400
Subject: [PATCH v1 2/2] Fix infinite loop in vacuum caused by visibility
mismatch
lazy_scan_prune() may loop infinitely if a tuple deemed non-removable by
heap_prune_satisfies_vacuum() in heap_page_prune() is later deemed
HEAPTUPLE_DEAD in lazy_scan_prune() by HeapTupleSatisfiesVacuum().
Both determine what transaction ID a tuple would be dead_after using
HeapTupleSatisfiesVacuumHorizon() and then compare that transaction ID
to a horizon. HeapTupleSatisfiesVacuum() uses OldestXmin from
VacuumCutoffs as that horizon and heap_prune_satisfies_vacuum() uses
GlobalVisTestIsRemovableXid() -- which checks the GlobalVisState that
was passed in to heap_page_prune().
OldestXmin is determined at the start of vacuuming the relation, but
GlobalVisState's maybe_needed value can change if a tuple's xmax falls
within GlobalVisState's fuzzy window and vacuum has had to take a new
snapshot since it started vacuuming the relation.
In rare cases, a backend's GlobalVisState->maybe_needed may move
backward -- for example if a standby with an older horizon which was not
connected at the start of the the vacuum later connects.
When this happends, a tuple with an xmax older than OldestXmin but newer
than the new value of maybe_needed after GlobalVisState was updated
could cause lazy_scan_prune() to loop infinitely.
By only computing the tuple's HTSV_Result once, in heap_page_prune(),
and saving it to be passed back to lazy_scan_prune(), we can avoid the
problem altogether. This also happens to be cheaper, since
HeapTupleSatisfiesVacuumHorizon() accesses the clog.
As a consequence, OldestXmin is now only used for determining whether or
not to freeze tuples and whether or not the whole page is visible -- not
for determining whether or not tuples are removable.
---
src/backend/access/heap/pruneheap.c | 47 ++++++++++++++++------------
src/backend/access/heap/vacuumlazy.c | 46 ++++++++++-----------------
src/include/access/heapam.h | 15 +++++++++
3 files changed, 58 insertions(+), 50 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 9f43bbe25f5..801cba51bc6 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -65,16 +65,6 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
-
- /*
- * Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
- *
- * Same indexing as ->marked.
- */
- int8 htsv[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
@@ -83,7 +73,7 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate);
+ PruneState *prstate, int8 *htsv);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -205,9 +195,10 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
{
int ndeleted,
nnewlpdead;
+ int8 htsv[MaxHeapTuplesPerPage + 1];
ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
- limited_ts, &nnewlpdead, NULL);
+ limited_ts, htsv, &nnewlpdead, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -254,6 +245,18 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* either have been set by TransactionIdLimitedForOldSnapshots, or
* InvalidTransactionId/0 respectively.
*
+ * htsv is an in/out parameter filled in by heap_page_prune() with tuple
+ * visibility, computed once for each tuple for correctness and efficiency
+ * reasons. Some callers want access to htsv for later decisions and metrics.
+ * htsv is of type int8[], instead of HTSV_Result[], so we can use -1 to
+ * indicate no visibility has been computed, e.g. for LP_DEAD items. It can be
+ * passed in uninitialized because all spots will be set either to a valid
+ * HTSV_Result value or -1.
+ *
+ * It is the caller's responsibility to ensure that htsv is
+ * MaxHeapTuplesPerPage + 1 in size. By making it MaxHeapTuplesPerPage + 1, we
+ * can avoid subtracting 1 everywhere even though FirstOffsetNumber is 1.
+ *
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
*
@@ -267,6 +270,7 @@ heap_page_prune(Relation relation, Buffer buffer,
GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
+ int8 htsv[MaxHeapTuplesPerPage + 1],
int *nnewlpdead,
OffsetNumber *off_loc)
{
@@ -332,7 +336,7 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Nothing to do if slot doesn't contain a tuple */
if (!ItemIdIsNormal(itemid))
{
- prstate.htsv[offnum] = -1;
+ htsv[offnum] = -1;
continue;
}
@@ -348,7 +352,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
+ htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
buffer);
}
@@ -373,7 +377,7 @@ heap_page_prune(Relation relation, Buffer buffer,
continue;
/* Process this item or chain of items */
- ndeleted += heap_prune_chain(buffer, offnum, &prstate);
+ ndeleted += heap_prune_chain(buffer, offnum, &prstate, htsv);
}
/* Clear the offset information once we have processed the given page. */
@@ -561,6 +565,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
* Prune specified line pointer or a HOT chain originating at line pointer.
*
+ * Tuple visibility information is provided in htsv.
+ *
* If the item is an index-referenced tuple (i.e. not a heap-only tuple),
* the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
* chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
@@ -588,7 +594,8 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
* Returns the number of tuples (to be) deleted from the page.
*/
static int
-heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
+heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate,
+ int8 *htsv)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -609,7 +616,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
if (ItemIdIsNormal(rootlp))
{
- Assert(prstate->htsv[rootoffnum] != -1);
+ Assert(htsv[rootoffnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
if (HeapTupleHeaderIsHeapOnly(htup))
@@ -632,7 +639,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* either here or while following a chain below. Whichever path
* gets there first will mark the tuple unused.
*/
- if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
+ if (htsv[rootoffnum] == HEAPTUPLE_DEAD &&
!HeapTupleHeaderIsHotUpdated(htup))
{
heap_prune_record_unused(prstate, rootoffnum);
@@ -700,7 +707,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
Assert(ItemIdIsNormal(lp));
- Assert(prstate->htsv[offnum] != -1);
+ Assert(htsv[offnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, lp);
/*
@@ -720,7 +727,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
tupdead = recent_dead = false;
- switch ((HTSV_Result) prstate->htsv[offnum])
+ switch ((HTSV_Result) htsv[offnum])
{
case HEAPTUPLE_DEAD:
tupdead = true;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2e61e2f20b4..8ba0da063db 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1526,12 +1526,14 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* of complexity just so we could deal with tuples that were DEAD to VACUUM,
* but nevertheless were left with storage after pruning.
*
- * The approach we take now is to restart pruning when the race condition is
- * detected. This allows heap_page_prune() to prune the tuples inserted by
- * the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * As of minor release blah, we circumvent this problem altogether by reusing
+ * the result of heap_page_prune()'s visibility check. Without the second call
+ * to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be
+ * no disagreement. We'll just handle such tuples as if they had become fully
+ * dead right after this operation completes instead of in the middle of it.
+ * Note that any tuple that becomes dead after the call to heap_page_prune()
+ * can't need to be frozen, because it was visible to another session when
+ * vacuum started.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1544,8 +1546,6 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- HeapTupleData tuple;
- HTSV_Result res;
int tuples_deleted,
lpdead_items,
live_tuples,
@@ -1556,6 +1556,7 @@ lazy_scan_prune(LVRelState *vacrel,
MultiXactId NewRelminMxid;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+ int8 htsv[MaxHeapTuplesPerPage + 1];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1566,8 +1567,6 @@ lazy_scan_prune(LVRelState *vacrel,
*/
maxoff = PageGetMaxOffsetNumber(page);
-retry:
-
/* Initialize (or reset) page-level state */
NewRelfrozenXid = vacrel->NewRelfrozenXid;
NewRelminMxid = vacrel->NewRelminMxid;
@@ -1586,7 +1585,7 @@ retry:
* that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
- InvalidTransactionId, 0, &nnewlpdead,
+ InvalidTransactionId, 0, htsv, &nnewlpdead,
&vacrel->offnum);
/*
@@ -1604,6 +1603,7 @@ retry:
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
+ HeapTupleHeader htup;
bool tuple_totally_frozen;
/*
@@ -1649,21 +1649,7 @@ retry:
Assert(ItemIdIsNormal(itemid));
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
- tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
- tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationGetRelid(rel);
-
- /*
- * DEAD tuples are almost always pruned into LP_DEAD line pointers by
- * heap_page_prune(), but it's possible that the tuple state changed
- * since heap_page_prune() looked. Handle that here by restarting.
- * (See comments at the top of function for a full explanation.)
- */
- res = HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf);
-
- if (unlikely(res == HEAPTUPLE_DEAD))
- goto retry;
+ htup = (HeapTupleHeader) PageGetItem(page, itemid);
/*
* The criteria for counting a tuple as live in this block need to
@@ -1684,7 +1670,7 @@ retry:
* (Cases where we bypass index vacuuming will violate this optimistic
* assumption, but the overall impact of that should be negligible.)
*/
- switch (res)
+ switch (htsv_get_valid_status(htsv[offnum]))
{
case HEAPTUPLE_LIVE:
@@ -1706,7 +1692,7 @@ retry:
{
TransactionId xmin;
- if (!HeapTupleHeaderXminCommitted(tuple.t_data))
+ if (!HeapTupleHeaderXminCommitted(htup))
{
prunestate->all_visible = false;
break;
@@ -1716,7 +1702,7 @@ retry:
* The inserter definitely committed. But is it old enough
* that everyone sees it as committed?
*/
- xmin = HeapTupleHeaderGetXmin(tuple.t_data);
+ xmin = HeapTupleHeaderGetXmin(htup);
if (!TransactionIdPrecedes(xmin, vacrel->OldestXmin))
{
prunestate->all_visible = false;
@@ -1772,7 +1758,7 @@ retry:
* now.
*/
prunestate->hastup = true; /* page makes rel truncation unsafe */
- if (heap_prepare_freeze_tuple(tuple.t_data,
+ if (heap_prepare_freeze_tuple(htup,
vacrel->relfrozenxid,
vacrel->relminmxid,
vacrel->FreezeLimit,
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index abf62d9df79..0b349427982 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -188,6 +188,7 @@ extern int heap_page_prune(Relation relation, Buffer buffer,
struct GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
+ int8 htsv[MaxHeapTuplesPerPage + 1],
int *nnewlpdead,
OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
@@ -230,4 +231,18 @@ extern bool ResolveCminCmaxDuringDecoding(struct HTAB *tuplecid_data,
extern void HeapCheckForSerializableConflictOut(bool valid, Relation relation, HeapTuple tuple,
Buffer buffer, Snapshot snapshot);
+/*
+ * Pruning calculates tuple visibility once and saves the results in an array
+ * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * guard against examining visibility status array members which have not yet
+ * been computed.
+ */
+static inline HTSV_Result
+htsv_get_valid_status(int status)
+{
+ Assert(status >= HEAPTUPLE_DEAD &&
+ status <= HEAPTUPLE_DELETE_IN_PROGRESS);
+ return (HTSV_Result) status;
+}
+
#endif /* HEAPAM_H */
--
2.34.1
REL15_v1-0001-Repro-for-vacuum-infinite-loop.patch (text/x-patch, 10.8K) download
From 5ecdf25ff6d2bab925a0ce9fe4a9c63873516e35 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 18 Jun 2024 16:09:00 -0400
Subject: [PATCH v1 1/2] Repro for vacuum infinite loop
This repro is not stable enough to be added as a test to the regression
suite. It is for demonstration purposes on the thread.
---
src/test/recovery/t/099_vacuum_hang.pl | 265 +++++++++++++++++++++++++
1 file changed, 265 insertions(+)
create mode 100644 src/test/recovery/t/099_vacuum_hang.pl
diff --git a/src/test/recovery/t/099_vacuum_hang.pl b/src/test/recovery/t/099_vacuum_hang.pl
new file mode 100644
index 00000000000..ab58182d04d
--- /dev/null
+++ b/src/test/recovery/t/099_vacuum_hang.pl
@@ -0,0 +1,265 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use IPC::Run qw(pump);
+use Time::HiRes qw(usleep);
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+my $tablespace1 = "test_vacuum_hang_tblspc";
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+hot_standby_feedback = on
+log_recovery_conflict_waits = true
+log_statement='all'
+log_connections=true
+log_lock_waits = true
+autovacuum = off
+maintenance_work_mem = 1024
+]);
+$node_primary->start;
+
+my $backup_name = 'my_backup';
+
+$node_primary->backup($backup_name);
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+$node_replica->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+my $orig_conninfo = $node_primary->connstr();
+
+# test schema / data
+my $table1 = "test_vacuum_hang_table";
+my $index1 = "test_vacuum_hang_index";
+my $col1 = "col1";
+
+my $psql_timeout = IPC::Run::timer(10);
+
+# Long-running Primary Session A
+my %psql_primaryA = ('stdin' => '', 'stdout' => '');
+$psql_primaryA{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryA{stdin},
+ \$psql_primaryA{stdout},
+ $psql_timeout);
+$psql_primaryA{stdout} = '';
+$psql_primaryA{stdin} = "set application_name=A;\n";
+
+# Long-running Primary Session B
+my %psql_primaryB = ('stdin' => '', 'stdout' => '');
+$psql_primaryB{run} =
+ $node_primary->background_psql($test_db, \$psql_primaryB{stdin},
+ \$psql_primaryB{stdout},
+ $psql_timeout);
+$psql_primaryB{stdout} = '';
+$psql_primaryB{stdin} = "set application_name=B;\n";
+
+# Long-running Replica Session A
+my %psql_replicaA = ('stdin' => '', 'stdout' => '');
+$psql_replicaA{run} =
+ $node_replica->background_psql($test_db, \$psql_replicaA{stdin},
+ \$psql_replicaA{stdout},
+ $psql_timeout);
+$psql_replicaA{stdout} = '';
+
+# Insert one tuple with value 1 which we can use to make sure the cursor has
+# successfully pinned and locked the buffer.
+#
+# Then insert and update enough rows that we force at least one round of index
+# vacuuming before getting to a dead tuple which was killed after the standby
+# is disconnected.
+#
+# Multiple index vacuuming passes is required to repro because after the
+# standby reconnects to the primary, our backend's GlobalVisStates will not
+# have been updated with the new horizon until an update is forced.
+#
+# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the
+# end of a round of index vacuuming, updating the backend's GlobalVisState
+# and, in our case, moving maybe_needed backwards.
+#
+# Then vacuum's first pass will continue and pruning will find our later
+# inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
+# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(${col1} int) with (autovacuum_enabled=false);
+ INSERT INTO $table1 VALUES (1);
+ INSERT INTO $table1 SELECT generate_series(2, 30000);
+ CREATE INDEX ${index1} on ${table1}(${col1});
+ UPDATE $table1 SET $col1 = 0 WHERE $col1 > 1;
+ UPDATE $table1 SET $col1 = 3 WHERE $col1 = 0;
+ UPDATE $table1 SET $col1 = 0 WHERE $col1 = 3;
+ UPDATE $table1 SET $col1 = 3 WHERE $col1 = 0;
+ UPDATE $table1 SET $col1 = 0 WHERE $col1 = 3;
+ UPDATE $table1 SET $col1 = 3 WHERE $col1 = 0;
+]);
+
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+my $walreceiver_pid = $node_replica->safe_psql($test_db, qq[
+ select pid from pg_stat_activity where backend_type = 'walreceiver';]);
+
+# Set primary_conninfo to something invalid on the replica and reload the
+# config. This will prevent the standby from reconnecting once the connection
+# is terminated. Then terminate the wal receiver. When a new WAL receiver
+# process starts up, it has to use the primary_conninfo to connect to the
+# primary and it will be unable to do so.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ SELECT pg_terminate_backend($walreceiver_pid)]);
+
+# Ensure the WAL receiver is no longer active on replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_activity where pid = $walreceiver_pid);] , 'f');
+
+# DECLARE and use a cursor on standby, causing the block of the relation to be
+# pinned and locked in a buffer on the replica. It is important that this is
+# after termination of the WAL receiver so that the primary does not know about
+# the cursor and it can't hold back the horizon on the primary.
+my $replica_cursor1 = "test_vacuum_hang_cursor_replica1";
+$psql_replicaA{stdin} .= qq[
+ BEGIN;
+ DECLARE $replica_cursor1 CURSOR FOR SELECT * FROM $table1;
+ FETCH FORWARD FROM $replica_cursor1;
+ ];
+
+# FETCH FORWARD should have returned a 1. That's how we know the cursor has a
+# pin and lock.
+ok(pump_until($psql_replicaA{run}, $psql_timeout,
+ \$psql_replicaA{stdout}, qr/^1$/m), "got first value from replica cursor");
+
+# Now insert and update a tuple which will be visible to the vacuum on the
+# primary but which will have xmax newer than the standby that was recently
+# disconnected.
+$psql_primaryA{stdin} .= qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET $col1 = 100 WHERE $col1 = 99;
+ SELECT 'after_update';
+ ];
+
+# Make sure the UPDATE finished
+ok(pump_until($psql_primaryA{run}, $psql_timeout,
+ \$psql_primaryA{stdout}, qr/^after_update$/m), "SELECT output from primary session A");
+
+# Open a cursor on the primary whose lock will keep VACUUM from getting a
+# cleanup lock on the first page of the relation. We want VACUUM to be able to
+# start, calculate initial values for OldestXmin and GlobalVisState and then be
+# unable to proceed with pruning our dead tuples. This will allow us to
+# reconnect the standby and push the horizon back before we start actual
+# pruning and vacuuming.
+my $primary_cursor1 = "test_vacuum_hang_cursor_primary1";
+$psql_primaryB{stdin} .= qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT * FROM $table1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ];
+
+# FETCH FORWARD should return a 1. That's how we know the cursor has a pin and
+# lock.
+ok(pump_until($psql_primaryB{run}, $psql_timeout,
+ \$psql_primaryB{stdout}, qr/^1$/m), "got first value from primary cursor");
+
+# Now start a VACUUM FREEZE on the primary. It will be unable to get a cleanup
+# lock and start pruning, so it will hang. It will call
+# vacuum_get_cutoffs() and establish values of OldestXmin and
+# GlobalVisState which are newer than all of our dead tuples.
+$psql_primaryA{stdin} .= qq[
+ VACUUM FREEZE $table1;
+ \\echo VACUUM
+ ];
+
+# Try and make sure our vacuum command has reached the server before we commit
+# the cursor.
+$psql_primaryA{run}->pump_nb();
+
+# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is just
+# waiting on the cleanup lock to start vacuuming. It needs to reach the state
+# of scanning heap. On 16+ we can easily ensure this by checking for
+# pg_stat_progress_vacuum phase 'scanning heap' with poll_query_until(). Here
+# just use a sleep. Maybe I could do something better with pump_until() but I
+# gave up.
+# We don't want the standby to re-establish a connection to the primary and
+# push the horizon back until we've saved initial values in GlobalVisState and
+# calculated OldestXmin.
+usleep(300_000);
+
+# Ensure the WAL receiver is still not active on the replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Allow the WAL receiver connection to re-establish. VACUUM is still
+# waiting for the ALTER INDEX to commit.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the VACUUM FREEZE proceeds and does its first round of
+# index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Commit the cursor so that the VACUUM can proceed.
+$psql_primaryB{stdin} .= qq[
+ COMMIT;
+ \\echo commit
+ ];
+
+ok(pump_until($psql_primaryB{run}, $psql_timeout,
+ \$psql_primaryB{stdout}, qr/^commit$/m), "CURSOR committing");
+
+$psql_primaryA{run}->pump_nb();
+
+# VACUUM proceeds with pruning and does a visibility check on each tuple. It
+# will find our final dead tuple non-removable (HEAPTUPLE_RECENTLY_DEAD) since
+# its xmax is after the new value of maybe_needed. Without the fix, after
+# pruning, in lazy_scan_prune(), vacuum does another visibility check, this
+# time with HeapTupleSatisfiesVacuum() which compares dead_after to OldestXmin.
+# It will find the tuple HEAPTUPLE_DEAD since its xmax precedes OldestXmin.
+# This will cause the infinite loop.
+pump $psql_primaryA{run} until ($psql_primaryA{stdout} =~ /VACUUM/ || $psql_timeout->is_expired);
+
+ok(!$psql_timeout->is_expired);
+
+# Commit the original cursor transaction on the replica so it can catch up. it
+# will end up replaying the VACUUM and not removing the tuple too.
+$psql_replicaA{stdin} .= qq[ COMMIT; ];
+
+$psql_replicaA{run}->pump_nb();
+
+$primary_lsn = $node_primary->lsn('flush');
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "insert into $table1 values (1);");
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Shut down psqls
+$psql_primaryA{stdin} .= "\\q\n";
+$psql_primaryA{run}->finish;
+
+$psql_primaryB{stdin} .= "\\q\n";
+$psql_primaryB{run}->finish;
+
+
+$psql_replicaA{stdin} .= "\\q\n";
+$psql_replicaA{run}->finish;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
REL16_v1-0001-Repro-for-vacuum-infinite-loop.patch (text/x-patch, 9.8K) download
From 0e80fcf56c24770409c9a9ad169af1efc8ec7ceb Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 19 Jun 2024 14:50:37 -0400
Subject: [PATCH v1 1/2] Repro for vacuum infinite loop
This repro is not stable enough to be added as a test to the regression
suite. It is for demonstration purposes on the thread.
---
src/test/recovery/t/099_vacuum_hang.pl | 254 +++++++++++++++++++++++++
1 file changed, 254 insertions(+)
create mode 100644 src/test/recovery/t/099_vacuum_hang.pl
diff --git a/src/test/recovery/t/099_vacuum_hang.pl b/src/test/recovery/t/099_vacuum_hang.pl
new file mode 100644
index 00000000000..3bd3b335e2c
--- /dev/null
+++ b/src/test/recovery/t/099_vacuum_hang.pl
@@ -0,0 +1,254 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use IPC::Run qw(pump);
+use Time::HiRes qw(usleep);
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+my $tablespace1 = "test_vacuum_hang_tblspc";
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+hot_standby_feedback = on
+log_recovery_conflict_waits = true
+log_statement='all'
+log_connections=true
+log_lock_waits = true
+autovacuum = off
+maintenance_work_mem = 1024
+]);
+$node_primary->start;
+
+my $backup_name = 'my_backup';
+
+$node_primary->backup($backup_name);
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+$node_replica->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+my $orig_conninfo = $node_primary->connstr();
+
+# test schema / data
+my $table1 = "test_vacuum_hang_table";
+my $index1 = "test_vacuum_hang_index";
+my $col1 = "col1";
+
+my $psql_timeout = IPC::Run::timer(15);
+
+# Long-running Primary Session A
+my $psql_primaryA =
+ $node_primary->background_psql($test_db, on_error_stop => 0);
+
+$psql_primaryA->query_safe("set application_name=A;");
+
+# Long-running Primary Session B
+my $psql_primaryB =
+ $node_primary->background_psql($test_db, on_error_stop => 0);
+
+$psql_primaryB->query_safe("set application_name=B;");
+
+
+# Long-running Replica Session A
+my $psql_replicaA =
+ $node_replica->background_psql($test_db, on_error_stop => 0);
+
+# Insert one tuple with value 1 which we can use to make sure the cursor has
+# successfully pinned and locked the buffer.
+#
+# Then insert and update enough rows that we force at least one round of index
+# vacuuming before getting to a dead tuple which was killed after the standby
+# is disconnected.
+#
+# Multiple index vacuuming passes is required to repro because after the
+# standby reconnects to the primary, our backend's GlobalVisStates will not
+# have been updated with the new horizon until an update is forced.
+#
+# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the
+# end of a round of index vacuuming, updating the backend's GlobalVisState
+# and, in our case, moving maybe_needed backwards.
+#
+# Then vacuum's first pass will continue and pruning will find our later
+# inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
+# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(${col1} int) with (autovacuum_enabled=false);
+ INSERT INTO $table1 VALUES (1);
+ INSERT INTO $table1 SELECT generate_series(2, 30000);
+ CREATE INDEX ${index1} on ${table1}(${col1});
+ UPDATE $table1 SET $col1 = 0 WHERE $col1 > 1;
+ UPDATE $table1 SET $col1 = 3 WHERE $col1 = 0;
+ UPDATE $table1 SET $col1 = 0 WHERE $col1 = 3;
+ UPDATE $table1 SET $col1 = 3 WHERE $col1 = 0;
+ UPDATE $table1 SET $col1 = 0 WHERE $col1 = 3;
+ UPDATE $table1 SET $col1 = 3 WHERE $col1 = 0;
+]);
+
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+my $walreceiver_pid = $node_replica->safe_psql($test_db, qq[
+ select pid from pg_stat_activity where backend_type = 'walreceiver';]);
+
+# Set primary_conninfo to something invalid on the replica and reload the
+# config. This will prevent the standby from reconnecting once the connection
+# is terminated. Then terminate the wal receiver. When a new WAL receiver
+# process starts up, it has to use the primary_conninfo to connect to the
+# primary and it will be unable to do so.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ SELECT pg_terminate_backend($walreceiver_pid)]);
+
+# Ensure the WAL receiver is no longer active on replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_activity where pid = $walreceiver_pid);] , 'f');
+
+# DECLARE and use a cursor on standby, causing the block of the relation to be
+# pinned and locked in a buffer on the replica. It is important that this is
+# after termination of the WAL receiver so that the primary does not know about
+# the cursor and it can't hold back the horizon on the primary.
+my $replica_cursor1 = "test_vacuum_hang_cursor_replica";
+
+# FETCH FORWARD should return a 1. That's how we know the cursor has a pin and
+# lock.
+$psql_replicaA->query_until(
+ qr/^1$/m,
+ qq[
+ BEGIN;
+ DECLARE $replica_cursor1 CURSOR FOR SELECT $col1 FROM $table1;
+ FETCH FORWARD FROM $replica_cursor1;
+ ]
+ );
+
+# Now insert and update a tuple which will be visible to the vacuum on the
+# primary but which will have xmax newer than the standby that was recently
+# disconnected.
+my $res = $psql_primaryA->query_safe(
+ qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET $col1 = 100 WHERE $col1 = 99;
+ SELECT 'after_update';
+ ]
+ );
+
+## Make sure the UPDATE finished
+like($res, qr/^after_update$/m, "UPDATE occurred on primary session A");
+
+# Open a cursor on the primary whose lock will keep VACUUM from getting a
+# cleanup lock on the first page of the relation. We want VACUUM to be able to
+# start, calculate initial values for OldestXmin and GlobalVisState and then be
+# unable to proceed with pruning our dead tuples. This will allow us to
+# reconnect the standby and push the horizon back before we start actual
+# pruning and vacuuming.
+my $primary_cursor1 = "test_vacuum_hang_cursor_primary";
+
+# FETCH FORWARD should return a 1. That's how we know the cursor has a pin and
+# lock.
+$res = $psql_primaryB->query_until(
+ qr/^1$/m,
+ qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT $col1 FROM $table1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ]
+ );
+
+# Now start a VACUUM FREEZE on the primary. It will be unable to get a cleanup
+# lock and start pruning, so it will hang. It will call
+# vacuum_get_cutoffs() and establish values of OldestXmin and
+# GlobalVisState which are newer than all of our dead tuples.
+$psql_primaryA->{stdin} .= qq[
+ VACUUM FREEZE $table1;
+ \\echo VACUUM
+ ];
+
+$psql_primaryA->{run}->pump_nb();
+
+# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is just
+# waiting on the cleanup lock to start vacuuming. pg_stat_progress_vacuum will
+# have been updated to phase 'scanning heap'. We don't want the standby to
+# re-establish a connection to the primary and push the horizon back until
+# we've saved initial values in GlobalVisState and calculated OldestXmin.
+# If for some reason this doesn't work, I found that throwing in a sleep helps.
+$node_primary->poll_query_until($test_db,
+ qq[ SELECT count(*) >= 1
+ FROM pg_stat_progress_vacuum
+ WHERE phase = 'scanning heap' AND
+ '${table1}'::regclass::oid = relid;
+ ] ,
+ 't');
+
+# Ensure the WAL receiver is still not active on the replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Allow the WAL receiver connection to re-establish.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the VACUUM FREEZE proceeds and does its first round of
+# index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Commit the cursor so that the VACUUM can proceed.
+$psql_primaryB->query_until(
+ qr/^commit$/m,
+ qq[
+ COMMIT;
+ \\echo commit
+ ]
+ );
+
+$psql_primaryA->{run}->pump_nb();
+
+# VACUUM proceeds with pruning and does a visibility check on each tuple. It
+# will find our final dead tuple non-removable (HEAPTUPLE_RECENTLY_DEAD) since
+# its xmax is after the new value of maybe_needed. Without the fix, after
+# pruning, in lazy_scan_prune(), vacuum does another visibility check, this
+# time with HeapTupleSatisfiesVacuum() which compares dead_after to OldestXmin.
+# It will find the tuple HEAPTUPLE_DEAD since its xmax precedes OldestXmin.
+# This will cause the infinite loop.
+pump $psql_primaryA->{run} until ($psql_primaryA->{stdout} =~ /VACUUM/ || $psql_timeout->is_expired);
+
+ok(!$psql_timeout->is_expired);
+
+# Commit the original cursor transaction on the replica so it can catch up. it
+# will end up replaying the VACUUM and not removing the tuple too.
+$psql_replicaA->query_safe( qq[ COMMIT; ] );
+
+$psql_replicaA->{run}->pump_nb();
+
+$primary_lsn = $node_primary->lsn('flush');
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);");
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+## Shut down psqls
+$psql_primaryA->quit;
+$psql_primaryB->quit;
+$psql_replicaA->quit;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-06-21 00:33 Melanie Plageman <melanieplageman@gmail.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 0 replies; 62+ messages in thread
From: Melanie Plageman @ 2024-06-21 00:33 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; Bowen Shi <zxwsbg12138@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 20, 2024 at 11:49 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:
>
> On Tue, Jun 18, 2024 at 6:51 PM Melanie Plageman
> <melanieplageman@gmail.com> wrote:
> >
> > Finally, upthread there is discussion of how we could end up doing a
> > catalog lookup after vacuum_get_cutoffs() and before the tuple
> > visibility check on 16. Assuming this is true, we would want to
> > backport the fix to 16 as well. I could use some help getting a repro
> > (using btree index deletion for example) of the infinite loop on 16.
>
> So, I ended up working on a new repro that works by forcing a round of
> index vacuuming after the standby reconnects and before pruning a dead
> tuple whose xmax is older than OldestXmin.
>
> At the end of the round of index vacuuming, _bt_pendingfsm_finalize()
> calls GetOldestNonRemovableTransactionId(), thereby updating the
> backend's GlobalVisState and moving maybe_needed backwards.
>
> Then vacuum's first pass will continue with pruning and find our later
> inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
> maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
>
> I make sure that the standby reconnects between vacuum_get_cutoffs()
> (vacuum_set_xid_limits() on 14/15) and pruning because I have a cursor
> on the page keeping VACUUM FREEZE from getting a cleanup lock.
>
> See the repros for step-by-step explanations of how it works.
>
> With this, I can repro the infinite loop on 14-16.
>
> Backporting 1ccc1e05ae fixes 16 but, with the new repro, 14 and 15
> error out with "cannot freeze committed xmax". I'm going to
> investigate further why this is happening. It definitely makes me
> wonder about the fix.
It turns out it was also erroring out on 16 (i.e. backporting
1ccc1e05ae did not fix anything), but I didn't notice it because the
perl TAP test passed. I also discovered we can hit this error in
master, so I started a thread about that here [1].
- Melanie
[1] https://www.postgresql.org/message-id/CAAKRu_bDD7oq9ZwB2OJqub5BovMG6UjEYsoK2LVttadjEqyRGg%40mail.gma...
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-06-25 19:37 Noah Misch <noah@leadboat.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-06-25 19:37 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 20, 2024 at 11:49:50AM -0400, Melanie Plageman wrote:
> On Tue, Jun 18, 2024 at 6:51 PM Melanie Plageman <melanieplageman@gmail.com> wrote:
> > I ended up manually backporting the logic from 1ccc1e05ae as opposed
> > to cherry-picking because it relied on a struct introduced in
> > 4e9fc3a9762065.
> Attached is the backport and repros for 15 and 16.
> When this happends, a tuple with an xmax older than OldestXmin but newer
s/happends/happens/
> @@ -700,7 +707,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> break;
>
> Assert(ItemIdIsNormal(lp));
> - Assert(prstate->htsv[offnum] != -1);
> + Assert(htsv[offnum] != -1);
The master patch removed this assert instead. Is the back branch non-removal
deliberate? If so, the patch is ready. I also confirm your results from your
repro recipe; thanks for including that.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-07-08 21:23 Melanie Plageman <melanieplageman@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 62+ messages in thread
From: Melanie Plageman @ 2024-07-08 21:23 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 25, 2024 at 3:37 PM Noah Misch <noah@leadboat.com> wrote:
>
> On Thu, Jun 20, 2024 at 11:49:50AM -0400, Melanie Plageman wrote:
> > On Tue, Jun 18, 2024 at 6:51 PM Melanie Plageman <melanieplageman@gmail.com> wrote:
> > > I ended up manually backporting the logic from 1ccc1e05ae as opposed
> > > to cherry-picking because it relied on a struct introduced in
> > > 4e9fc3a9762065.
>
> > Attached is the backport and repros for 15 and 16.
I think we are going with the fix proposed for master [1] which
compares dead_after to OldestXmin before using GlobalVisState.
Backporting 1ccc1e05ae doesn't actually fix the problem. We just end
up erroring out when attempting to freeze the tuple we didn't remove.
As such, attached is my proposed fix for affected stable branches. It
is based off of the fix proposed in [1] but is a bit different in each
version due to surrounding code changes.
The test I added passes locally and on linux and windows in CI (on 15+
which have CI). I don't have enough cirrus credits to run the tests on
mac. I am nervous about the test flaking on the buildfarm. But I did
the best I could to try to make it stable. I think keeping it as a
separate commit should be easiest in case we have to revert it?
Thanks to Heikki for backporting BackgroundPsql -- this made my life
much easier!!
- Melanie
[1] https://www.postgresql.org/message-id/CAAKRu_Z4PybtZ0i_NKOr-vbrFW5p1ZdfEfUqaeU8fLPhszpP_g%40mail.gma...
Attachments: download all (8 files, .tar.gz)
REL14_v2-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch (text/x-patch, 9.4K) download
From 079d6460300aed73beb5bbbd576158ac834c86da Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 2 Jul 2024 21:47:47 -0400
Subject: [PATCH v2 1/2] Test that vacuum removes tuples older than OldestXmin
If vacuum fails to prune a tuple killed before OldestXmin, it will later
find that tuple dead in lazy_scan_prune() and loop infinitely.
---
.../recovery/t/043_vacuum_horizon_floor.pl | 232 ++++++++++++++++++
1 file changed, 232 insertions(+)
create mode 100644 src/test/recovery/t/043_vacuum_horizon_floor.pl
diff --git a/src/test/recovery/t/043_vacuum_horizon_floor.pl b/src/test/recovery/t/043_vacuum_horizon_floor.pl
new file mode 100644
index 00000000000..5da30f0a3fd
--- /dev/null
+++ b/src/test/recovery/t/043_vacuum_horizon_floor.pl
@@ -0,0 +1,232 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# Test that vacuum prunes away all dead tuples killed before OldestXmin
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+hot_standby_feedback = on
+autovacuum = off
+maintenance_work_mem = 1024
+]);
+$node_primary->start;
+
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+
+$node_primary->backup('my_backup');
+$node_replica->init_from_backup($node_primary, 'my_backup',
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+# Save the original connection info for later use
+my $orig_conninfo = $node_primary->connstr();
+
+my $table1 = "vac_horizon_floor_table";
+
+# Long-running Primary Session A
+my $psql_primaryA =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Long-running Primary Session B
+my $psql_primaryB =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Insert and update enough rows that we force at least one round of index
+# vacuuming before getting to a dead tuple which was killed after the standby
+# is disconnected.
+#
+# We need multiple index vacuuming passes to repro because after the standby
+# reconnects to the primary, our backend's GlobalVisStates will not have been
+# updated with the new horizon until something forces them to be updated.
+#
+# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the
+# end of a round of index vacuuming, updating the backend's GlobalVisState
+# and, in our case, moving maybe_needed backwards.
+#
+# Then vacuum's first pass will continue and pruning will find our later
+# inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
+# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(col1 int) with (autovacuum_enabled=false);
+ INSERT INTO $table1 SELECT generate_series(1, 200000);
+ CREATE INDEX on ${table1}(col1);
+ UPDATE $table1 SET col1 = 0 WHERE col1 > 1;
+ INSERT INTO $table1 VALUES(1);
+]);
+
+# We will later move the primary forward while the standby is disconnected. For
+# now, however, there is no reason not to wait for the standby to catch up.
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Test that the WAL receiver is up and running.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Set primary_conninfo to something invalid on the replica and reload the
+# config. Once the config is reloaded, the startup process will force the WAL
+# receiver to restart and it will be unable to reconnect because of the invalid
+# connection information.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ ]);
+
+# Wait until the WAL receiver has shut down and been unable to start up again.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Now insert and update a tuple which will be visible to the vacuum on the
+# primary but which will have xmax newer than the oldest xmin on the standby
+# that was recently disconnected.
+my $res = $psql_primaryA->query_safe(
+ qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET col1 = 100 WHERE col1 = 99;
+ SELECT 'after_update';
+ ]
+ );
+
+# Make sure the UPDATE finished
+like($res, qr/^after_update$/m, "UPDATE occurred on primary session A");
+
+# Open a cursor on the primary whose pin will keep VACUUM from getting a
+# cleanup lock on the first page of the relation. We want VACUUM to be able to
+# start, calculate initial values for OldestXmin and GlobalVisState and then be
+# unable to proceed with pruning our dead tuples. This will allow us to
+# reconnect the standby and push the horizon back before we start actual
+# pruning and vacuuming.
+my $primary_cursor1 = "vac_horizon_floor_cursor";
+
+# The first value inserted into the table was a 1, so FETCH FORWARD should
+# return a 1. That's how we know the cursor has a pin.
+$res = $psql_primaryB->query_safe(
+ qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT col1 FROM $table1 WHERE col1 = 1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ]
+ );
+
+is($res, 1, qq[Cursor query returned $res. Expected value 1.]);
+
+# Get the PID of the session which will run the VACUUM FREEZE so that we can
+# use it to filter pg_stat_activity later.
+my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();");
+
+# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs()
+# and establish values of OldestXmin and GlobalVisState which are newer than
+# all of our dead tuples. Then it will be unable to get a cleanup lock to start
+# pruning, so it will hang.
+$psql_primaryA->{stdin} .= qq[
+ VACUUM FREEZE $table1;
+ \\echo VACUUM
+ ];
+
+# Make sure the VACUUM command makes it to the server.
+$psql_primaryA->{run}->pump_nb();
+
+# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is just
+# waiting on the lock to start vacuuming. We don't want the standby to
+# re-establish a connection to the primary and push the horizon back until
+# we've saved initial values in GlobalVisState and calculated OldestXmin.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT count(*) >= 1 FROM pg_stat_activity
+ WHERE pid = $vacuum_pid
+ AND wait_event = 'BufferPin';
+ ],
+ 't');
+
+# Ensure the WAL receiver is still not active on the replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Allow the WAL receiver connection to re-establish.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+# Ensure the new WAL receiver has connected.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the VACUUM FREEZE proceeds and does its first round of
+# index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Move the cursor forward to the next 1. We inserted the 1 much later, so
+# advancing the cursor should allow vacuum to proceed vacuuming most pages of
+# the relation. Because we set maintanence_work_mem sufficiently low, we expect
+# that a round of index vacuuming has happened and that the vacuum is now
+# waiting for the cursor to release its pin on the last page of the relation.
+$res = $psql_primaryB->query_safe("FETCH $primary_cursor1");
+is($res, 1, qq[Cursor query returned $res from second fetch. Expected value 1.]);
+
+# Prevent the test from incorrectly passing by confirming that we did indeed do
+# a pass of index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ SELECT index_vacuum_count > 0
+ FROM pg_stat_progress_vacuum
+ WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass;
+ ] , 't');
+
+# Commit the cursor so that the VACUUM can complete.
+$psql_primaryB->query_until(
+ qr/^commit$/m,
+ qq[
+ COMMIT;
+ \\echo commit
+ ]
+ );
+
+# VACUUM proceeds with pruning and does a visibility check on each tuple. In
+# older versions of Postgres, pruning found our final dead tuple
+# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new
+# value of maybe_needed. Then lazy_scan_prune() would infinitely loop
+# because HeapTupleSatisfiesVacuum() would find the tuple HEAPTUPLE_DEAD
+# because its xmax preceded OldestXmin. This was fixed by passing OldestXmin
+# to heap_page_prune() and removing all tuples whose xmaxes precede
+# OldestXmin.
+#
+# With the fix, VACUUM should finish successfully, incrementing the table
+# vacuum_count.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT vacuum_count > 0
+ FROM pg_stat_all_tables WHERE relname = '${table1}';
+ ]
+ , 't');
+
+$primary_lsn = $node_primary->lsn('flush');
+
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);");
+
+# Nothing on the replica should cause a recovery conflict, so this should
+# finish successfully.
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+## Shut down psqls
+$psql_primaryA->quit;
+$psql_primaryB->quit;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
REL15_v2-0002-Ensure-vacuum-removes-all-visibly-dead-tuples-old.patch (text/x-patch, 6.5K) download
From 235c072eb1c5f01d8da6c4cd2d1306abc3d87e14 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 8 Jul 2024 16:54:35 -0400
Subject: [PATCH v2 2/2] Ensure vacuum removes all visibly dead tuples older
than OldestXmin
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than GlobalVisState->maybe_needed,
it will loop infinitely in lazy_scan_prune(), which compares tuples'
visibility information to OldestXmin.
Starting in version 14, which uses GlobalVisState for visibility testing
during pruning, it is possible for GlobalVisState->maybe_needed to
precede OldestXmin if maybe_needed is forced to go backward while vacuum
is running. This can happen if a disconnected standby with a running
transaction older than VacuumCutoffs->OldestXmin reconnects to the
primary after vacuum initially calculates GlobalVisState and OldestXmin.
Fix this by having vacuum always remove tuples older than OldestXmin
during pruning. This is okay because the standby won't replay the tuple
removal until the tuple is removable. Thus, the worst that can happen is
a recovery conflict.
Fixes BUG# 17257
Back-patched in versions 14-17
Author: Melanie Plageman
Reviewed-by: Noah Misch
Discussion: https://postgre.es/m/CAAKRu_Y_NJzF4-8gzTTeaOuUL3CcGoXPjXcAHbTTygT8AyVqag%40mail.gmail.com
---
src/backend/access/heap/pruneheap.c | 48 +++++++++++++++++++++-------
src/backend/access/heap/vacuumlazy.c | 2 +-
src/include/access/heapam.h | 1 +
3 files changed, 38 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 9f43bbe25f5..555d7c6a1e3 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -33,7 +33,8 @@ typedef struct
{
Relation rel;
- /* tuple visibility test, initialized for the relation */
+ /* State used to test tuple visibility; Initialized for the relation */
+ TransactionId oldest_xmin;
GlobalVisState *vistest;
/*
@@ -206,7 +207,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int ndeleted,
nnewlpdead;
- ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
+ ndeleted = heap_page_prune(relation, buffer, InvalidTransactionId,
+ vistest, limited_xmin,
limited_ts, &nnewlpdead, NULL);
/*
@@ -248,11 +250,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* also need to account for a reduction in the length of the line pointer
* array following array truncation by us.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum). old_snap_xmin / old_snap_ts need to
- * either have been set by TransactionIdLimitedForOldSnapshots, or
- * InvalidTransactionId/0 respectively.
+ * vistest and oldest_xmin are used to distinguish whether tuples are DEAD or
+ * RECENTLY_DEAD (see heap_prune_satisfies_vacuum and
+ * HeapTupleSatisfiesVacuum). If oldest_xmin is provided by the caller, it is
+ * used before consulting GlobalVisState.
+ *
+ * old_snap_xmin / old_snap_ts need to either have been set by
+ * TransactionIdLimitedForOldSnapshots, or InvalidTransactionId/0 respectively.
*
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
@@ -264,6 +268,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
int
heap_page_prune(Relation relation, Buffer buffer,
+ TransactionId oldest_xmin,
GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
@@ -291,6 +296,7 @@ heap_page_prune(Relation relation, Buffer buffer,
*/
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
+ prstate.oldest_xmin = oldest_xmin;
prstate.vistest = vistest;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
@@ -520,13 +526,31 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
/*
- * First check if GlobalVisTestIsRemovableXid() is sufficient to find the
- * row dead. If not, and old_snapshot_threshold is enabled, try to use the
- * lowered horizon.
+ * For VACUUM, we must be sure to prune tuples with xmax older than
+ * oldest_xmin -- a visibility cutoff determined at the beginning of
+ * vacuuming the relation. oldest_xmin is used for freezing determination
+ * and we cannot freeze dead tuples' xmaxes.
+ */
+ if (TransactionIdIsValid(prstate->oldest_xmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * Determine whether or not the tuple is considered dead when compared
+ * with the provided GlobalVisState. On-access pruning does not provide
+ * oldest_xmin. And for vacuum, even if the tuple's xmax is not older than
+ * oldest_xmin, GlobalVisTestIsRemovableXid() could find the row dead if
+ * the GlobalVisState has been updated since the beginning of vacuuming
+ * the relation.
*/
if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
- res = HEAPTUPLE_DEAD;
- else if (OldSnapshotThresholdActive())
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * If GlobalVisTestIsRemovableXid() is not sufficient to find the row dead
+ * and old_snapshot_threshold is enabled, try to use the lowered horizon.
+ */
+ if (OldSnapshotThresholdActive())
{
/* haven't determined limited horizon yet, requests */
if (!TransactionIdIsValid(prstate->old_snap_xmin))
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2e61e2f20b4..f243dc49c96 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1585,7 +1585,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->OldestXmin, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index abf62d9df79..50b9e729e63 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -185,6 +185,7 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern int heap_page_prune(Relation relation, Buffer buffer,
+ TransactionId oldest_xmin,
struct GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
--
2.34.1
REL14_v2-0002-Ensure-vacuum-removes-all-visibly-dead-tuples-old.patch (text/x-patch, 6.5K) download
From 3f19417bc917f6ea20992a61fde443d04143dc1d Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 8 Jul 2024 16:42:33 -0400
Subject: [PATCH v2 2/2] Ensure vacuum removes all visibly dead tuples older
than OldestXmin
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than GlobalVisState->maybe_needed,
it will loop infinitely in lazy_scan_prune(), which compares tuples'
visibility information to OldestXmin.
Starting in version 14, which uses GlobalVisState for visibility testing
during pruning, it is possible for GlobalVisState->maybe_needed to
precede OldestXmin if maybe_needed is forced to go backward while vacuum
is running. This can happen if a disconnected standby with a running
transaction older than VacuumCutoffs->OldestXmin reconnects to the
primary after vacuum initially calculates GlobalVisState and OldestXmin.
Fix this by having vacuum always remove tuples older than OldestXmin
during pruning. This is okay because the standby won't replay the tuple
removal until the tuple is removable. Thus, the worst that can happen is
a recovery conflict.
Fixes BUG# 17257
Back-patched in versions 14-17
Author: Melanie Plageman
Reviewed-by: Noah Misch
Discussion: https://postgre.es/m/CAAKRu_Y_NJzF4-8gzTTeaOuUL3CcGoXPjXcAHbTTygT8AyVqag%40mail.gmail.com
---
src/backend/access/heap/pruneheap.c | 47 +++++++++++++++++++++-------
src/backend/access/heap/vacuumlazy.c | 2 +-
src/include/access/heapam.h | 1 +
3 files changed, 37 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index f7f8056d631..0856bb5f302 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -32,7 +32,8 @@ typedef struct
{
Relation rel;
- /* tuple visibility test, initialized for the relation */
+ /* State used to test tuple visibility; Initialized for the relation */
+ TransactionId oldest_xmin;
GlobalVisState *vistest;
/*
@@ -202,7 +203,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
/* OK to prune */
- (void) heap_page_prune(relation, buffer, vistest,
+ (void) heap_page_prune(relation, buffer, InvalidTransactionId, vistest,
limited_xmin, limited_ts,
true, NULL);
}
@@ -218,11 +219,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*
* Caller must have pin and buffer cleanup lock on the page.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum). old_snap_xmin / old_snap_ts need to
- * either have been set by TransactionIdLimitedForOldSnapshots, or
- * InvalidTransactionId/0 respectively.
+ * vistest and oldest_xmin are used to distinguish whether tuples are DEAD or
+ * RECENTLY_DEAD (see heap_prune_satisfies_vacuum and
+ * HeapTupleSatisfiesVacuum). If oldest_xmin is provided by the caller, it is
+ * used before consulting GlobalVisState.
+ *
+ * old_snap_xmin / old_snap_ts need to either have been set by
+ * TransactionIdLimitedForOldSnapshots, or InvalidTransactionId/0 respectively.
*
* If report_stats is true then we send the number of reclaimed heap-only
* tuples to pgstats. (This must be false during vacuum, since vacuum will
@@ -236,6 +239,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
int
heap_page_prune(Relation relation, Buffer buffer,
+ TransactionId oldest_xmin,
GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
@@ -262,6 +266,7 @@ heap_page_prune(Relation relation, Buffer buffer,
*/
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
+ prstate.oldest_xmin = oldest_xmin;
prstate.vistest = vistest;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
@@ -512,13 +517,31 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
/*
- * First check if GlobalVisTestIsRemovableXid() is sufficient to find the
- * row dead. If not, and old_snapshot_threshold is enabled, try to use the
- * lowered horizon.
+ * For VACUUM, we must be sure to prune tuples with xmax older than
+ * oldest_xmin -- a visibility cutoff determined at the beginning of
+ * vacuuming the relation. oldest_xmin is used for freezing determination
+ * and we cannot freeze dead tuples' xmaxes.
+ */
+ if (TransactionIdIsValid(prstate->oldest_xmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * Determine whether or not the tuple is considered dead when compared
+ * with the provided GlobalVisState. On-access pruning does not provide
+ * oldest_xmin. And for vacuum, even if the tuple's xmax is not older than
+ * oldest_xmin, GlobalVisTestIsRemovableXid() could find the row dead if
+ * the GlobalVisState has been updated since the beginning of vacuuming
+ * the relation.
*/
if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
- res = HEAPTUPLE_DEAD;
- else if (OldSnapshotThresholdActive())
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * If GlobalVisTestIsRemovableXid() is not sufficient to find the row dead
+ * and old_snapshot_threshold is enabled, try to use the lowered horizon.
+ */
+ if (OldSnapshotThresholdActive())
{
/* haven't determined limited horizon yet, requests */
if (!TransactionIdIsValid(prstate->old_snap_xmin))
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8aab6e324e0..5898eae127f 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1738,7 +1738,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->OldestXmin, vistest,
InvalidTransactionId, 0, false,
&vacrel->offnum);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 4f1dff9ca1b..b484db8fa57 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -186,6 +186,7 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern int heap_page_prune(Relation relation, Buffer buffer,
+ TransactionId oldest_xmin,
struct GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts_ts,
--
2.34.1
REL15_v2-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch (text/x-patch, 9.4K) download
From 917e1e39ebf0b3d66c4f092beb009c1ea103c071 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 2 Jul 2024 21:47:47 -0400
Subject: [PATCH v2 1/2] Test that vacuum removes tuples older than OldestXmin
If vacuum fails to prune a tuple killed before OldestXmin, it will later
find that tuple dead in lazy_scan_prune() and loop infinitely.
---
.../recovery/t/043_vacuum_horizon_floor.pl | 232 ++++++++++++++++++
1 file changed, 232 insertions(+)
create mode 100644 src/test/recovery/t/043_vacuum_horizon_floor.pl
diff --git a/src/test/recovery/t/043_vacuum_horizon_floor.pl b/src/test/recovery/t/043_vacuum_horizon_floor.pl
new file mode 100644
index 00000000000..5da30f0a3fd
--- /dev/null
+++ b/src/test/recovery/t/043_vacuum_horizon_floor.pl
@@ -0,0 +1,232 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# Test that vacuum prunes away all dead tuples killed before OldestXmin
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+hot_standby_feedback = on
+autovacuum = off
+maintenance_work_mem = 1024
+]);
+$node_primary->start;
+
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+
+$node_primary->backup('my_backup');
+$node_replica->init_from_backup($node_primary, 'my_backup',
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+# Save the original connection info for later use
+my $orig_conninfo = $node_primary->connstr();
+
+my $table1 = "vac_horizon_floor_table";
+
+# Long-running Primary Session A
+my $psql_primaryA =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Long-running Primary Session B
+my $psql_primaryB =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Insert and update enough rows that we force at least one round of index
+# vacuuming before getting to a dead tuple which was killed after the standby
+# is disconnected.
+#
+# We need multiple index vacuuming passes to repro because after the standby
+# reconnects to the primary, our backend's GlobalVisStates will not have been
+# updated with the new horizon until something forces them to be updated.
+#
+# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the
+# end of a round of index vacuuming, updating the backend's GlobalVisState
+# and, in our case, moving maybe_needed backwards.
+#
+# Then vacuum's first pass will continue and pruning will find our later
+# inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
+# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(col1 int) with (autovacuum_enabled=false);
+ INSERT INTO $table1 SELECT generate_series(1, 200000);
+ CREATE INDEX on ${table1}(col1);
+ UPDATE $table1 SET col1 = 0 WHERE col1 > 1;
+ INSERT INTO $table1 VALUES(1);
+]);
+
+# We will later move the primary forward while the standby is disconnected. For
+# now, however, there is no reason not to wait for the standby to catch up.
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Test that the WAL receiver is up and running.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Set primary_conninfo to something invalid on the replica and reload the
+# config. Once the config is reloaded, the startup process will force the WAL
+# receiver to restart and it will be unable to reconnect because of the invalid
+# connection information.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ ]);
+
+# Wait until the WAL receiver has shut down and been unable to start up again.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Now insert and update a tuple which will be visible to the vacuum on the
+# primary but which will have xmax newer than the oldest xmin on the standby
+# that was recently disconnected.
+my $res = $psql_primaryA->query_safe(
+ qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET col1 = 100 WHERE col1 = 99;
+ SELECT 'after_update';
+ ]
+ );
+
+# Make sure the UPDATE finished
+like($res, qr/^after_update$/m, "UPDATE occurred on primary session A");
+
+# Open a cursor on the primary whose pin will keep VACUUM from getting a
+# cleanup lock on the first page of the relation. We want VACUUM to be able to
+# start, calculate initial values for OldestXmin and GlobalVisState and then be
+# unable to proceed with pruning our dead tuples. This will allow us to
+# reconnect the standby and push the horizon back before we start actual
+# pruning and vacuuming.
+my $primary_cursor1 = "vac_horizon_floor_cursor";
+
+# The first value inserted into the table was a 1, so FETCH FORWARD should
+# return a 1. That's how we know the cursor has a pin.
+$res = $psql_primaryB->query_safe(
+ qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT col1 FROM $table1 WHERE col1 = 1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ]
+ );
+
+is($res, 1, qq[Cursor query returned $res. Expected value 1.]);
+
+# Get the PID of the session which will run the VACUUM FREEZE so that we can
+# use it to filter pg_stat_activity later.
+my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();");
+
+# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs()
+# and establish values of OldestXmin and GlobalVisState which are newer than
+# all of our dead tuples. Then it will be unable to get a cleanup lock to start
+# pruning, so it will hang.
+$psql_primaryA->{stdin} .= qq[
+ VACUUM FREEZE $table1;
+ \\echo VACUUM
+ ];
+
+# Make sure the VACUUM command makes it to the server.
+$psql_primaryA->{run}->pump_nb();
+
+# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is just
+# waiting on the lock to start vacuuming. We don't want the standby to
+# re-establish a connection to the primary and push the horizon back until
+# we've saved initial values in GlobalVisState and calculated OldestXmin.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT count(*) >= 1 FROM pg_stat_activity
+ WHERE pid = $vacuum_pid
+ AND wait_event = 'BufferPin';
+ ],
+ 't');
+
+# Ensure the WAL receiver is still not active on the replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Allow the WAL receiver connection to re-establish.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+# Ensure the new WAL receiver has connected.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the VACUUM FREEZE proceeds and does its first round of
+# index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Move the cursor forward to the next 1. We inserted the 1 much later, so
+# advancing the cursor should allow vacuum to proceed vacuuming most pages of
+# the relation. Because we set maintanence_work_mem sufficiently low, we expect
+# that a round of index vacuuming has happened and that the vacuum is now
+# waiting for the cursor to release its pin on the last page of the relation.
+$res = $psql_primaryB->query_safe("FETCH $primary_cursor1");
+is($res, 1, qq[Cursor query returned $res from second fetch. Expected value 1.]);
+
+# Prevent the test from incorrectly passing by confirming that we did indeed do
+# a pass of index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ SELECT index_vacuum_count > 0
+ FROM pg_stat_progress_vacuum
+ WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass;
+ ] , 't');
+
+# Commit the cursor so that the VACUUM can complete.
+$psql_primaryB->query_until(
+ qr/^commit$/m,
+ qq[
+ COMMIT;
+ \\echo commit
+ ]
+ );
+
+# VACUUM proceeds with pruning and does a visibility check on each tuple. In
+# older versions of Postgres, pruning found our final dead tuple
+# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new
+# value of maybe_needed. Then lazy_scan_prune() would infinitely loop
+# because HeapTupleSatisfiesVacuum() would find the tuple HEAPTUPLE_DEAD
+# because its xmax preceded OldestXmin. This was fixed by passing OldestXmin
+# to heap_page_prune() and removing all tuples whose xmaxes precede
+# OldestXmin.
+#
+# With the fix, VACUUM should finish successfully, incrementing the table
+# vacuum_count.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT vacuum_count > 0
+ FROM pg_stat_all_tables WHERE relname = '${table1}';
+ ]
+ , 't');
+
+$primary_lsn = $node_primary->lsn('flush');
+
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);");
+
+# Nothing on the replica should cause a recovery conflict, so this should
+# finish successfully.
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+## Shut down psqls
+$psql_primaryA->quit;
+$psql_primaryB->quit;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
REL16_v2-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch (text/x-patch, 9.4K) download
From 8aa4cd77bc9116ed88fad14a8a9b2e1f8d511437 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 2 Jul 2024 21:47:47 -0400
Subject: [PATCH v2 1/2] Test that vacuum removes tuples older than OldestXmin
If vacuum fails to prune a tuple killed before OldestXmin, it will later
find that tuple dead in lazy_scan_prune() and loop infinitely.
---
.../recovery/t/043_vacuum_horizon_floor.pl | 232 ++++++++++++++++++
1 file changed, 232 insertions(+)
create mode 100644 src/test/recovery/t/043_vacuum_horizon_floor.pl
diff --git a/src/test/recovery/t/043_vacuum_horizon_floor.pl b/src/test/recovery/t/043_vacuum_horizon_floor.pl
new file mode 100644
index 00000000000..5da30f0a3fd
--- /dev/null
+++ b/src/test/recovery/t/043_vacuum_horizon_floor.pl
@@ -0,0 +1,232 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# Test that vacuum prunes away all dead tuples killed before OldestXmin
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+hot_standby_feedback = on
+autovacuum = off
+maintenance_work_mem = 1024
+]);
+$node_primary->start;
+
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+
+$node_primary->backup('my_backup');
+$node_replica->init_from_backup($node_primary, 'my_backup',
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+# Save the original connection info for later use
+my $orig_conninfo = $node_primary->connstr();
+
+my $table1 = "vac_horizon_floor_table";
+
+# Long-running Primary Session A
+my $psql_primaryA =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Long-running Primary Session B
+my $psql_primaryB =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Insert and update enough rows that we force at least one round of index
+# vacuuming before getting to a dead tuple which was killed after the standby
+# is disconnected.
+#
+# We need multiple index vacuuming passes to repro because after the standby
+# reconnects to the primary, our backend's GlobalVisStates will not have been
+# updated with the new horizon until something forces them to be updated.
+#
+# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the
+# end of a round of index vacuuming, updating the backend's GlobalVisState
+# and, in our case, moving maybe_needed backwards.
+#
+# Then vacuum's first pass will continue and pruning will find our later
+# inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
+# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(col1 int) with (autovacuum_enabled=false);
+ INSERT INTO $table1 SELECT generate_series(1, 200000);
+ CREATE INDEX on ${table1}(col1);
+ UPDATE $table1 SET col1 = 0 WHERE col1 > 1;
+ INSERT INTO $table1 VALUES(1);
+]);
+
+# We will later move the primary forward while the standby is disconnected. For
+# now, however, there is no reason not to wait for the standby to catch up.
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Test that the WAL receiver is up and running.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Set primary_conninfo to something invalid on the replica and reload the
+# config. Once the config is reloaded, the startup process will force the WAL
+# receiver to restart and it will be unable to reconnect because of the invalid
+# connection information.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ ]);
+
+# Wait until the WAL receiver has shut down and been unable to start up again.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Now insert and update a tuple which will be visible to the vacuum on the
+# primary but which will have xmax newer than the oldest xmin on the standby
+# that was recently disconnected.
+my $res = $psql_primaryA->query_safe(
+ qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET col1 = 100 WHERE col1 = 99;
+ SELECT 'after_update';
+ ]
+ );
+
+# Make sure the UPDATE finished
+like($res, qr/^after_update$/m, "UPDATE occurred on primary session A");
+
+# Open a cursor on the primary whose pin will keep VACUUM from getting a
+# cleanup lock on the first page of the relation. We want VACUUM to be able to
+# start, calculate initial values for OldestXmin and GlobalVisState and then be
+# unable to proceed with pruning our dead tuples. This will allow us to
+# reconnect the standby and push the horizon back before we start actual
+# pruning and vacuuming.
+my $primary_cursor1 = "vac_horizon_floor_cursor";
+
+# The first value inserted into the table was a 1, so FETCH FORWARD should
+# return a 1. That's how we know the cursor has a pin.
+$res = $psql_primaryB->query_safe(
+ qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT col1 FROM $table1 WHERE col1 = 1;
+ FETCH FORWARD FROM $primary_cursor1;
+ ]
+ );
+
+is($res, 1, qq[Cursor query returned $res. Expected value 1.]);
+
+# Get the PID of the session which will run the VACUUM FREEZE so that we can
+# use it to filter pg_stat_activity later.
+my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();");
+
+# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs()
+# and establish values of OldestXmin and GlobalVisState which are newer than
+# all of our dead tuples. Then it will be unable to get a cleanup lock to start
+# pruning, so it will hang.
+$psql_primaryA->{stdin} .= qq[
+ VACUUM FREEZE $table1;
+ \\echo VACUUM
+ ];
+
+# Make sure the VACUUM command makes it to the server.
+$psql_primaryA->{run}->pump_nb();
+
+# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is just
+# waiting on the lock to start vacuuming. We don't want the standby to
+# re-establish a connection to the primary and push the horizon back until
+# we've saved initial values in GlobalVisState and calculated OldestXmin.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT count(*) >= 1 FROM pg_stat_activity
+ WHERE pid = $vacuum_pid
+ AND wait_event = 'BufferPin';
+ ],
+ 't');
+
+# Ensure the WAL receiver is still not active on the replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Allow the WAL receiver connection to re-establish.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+# Ensure the new WAL receiver has connected.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the VACUUM FREEZE proceeds and does its first round of
+# index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Move the cursor forward to the next 1. We inserted the 1 much later, so
+# advancing the cursor should allow vacuum to proceed vacuuming most pages of
+# the relation. Because we set maintanence_work_mem sufficiently low, we expect
+# that a round of index vacuuming has happened and that the vacuum is now
+# waiting for the cursor to release its pin on the last page of the relation.
+$res = $psql_primaryB->query_safe("FETCH $primary_cursor1");
+is($res, 1, qq[Cursor query returned $res from second fetch. Expected value 1.]);
+
+# Prevent the test from incorrectly passing by confirming that we did indeed do
+# a pass of index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ SELECT index_vacuum_count > 0
+ FROM pg_stat_progress_vacuum
+ WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass;
+ ] , 't');
+
+# Commit the cursor so that the VACUUM can complete.
+$psql_primaryB->query_until(
+ qr/^commit$/m,
+ qq[
+ COMMIT;
+ \\echo commit
+ ]
+ );
+
+# VACUUM proceeds with pruning and does a visibility check on each tuple. In
+# older versions of Postgres, pruning found our final dead tuple
+# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new
+# value of maybe_needed. Then lazy_scan_prune() would infinitely loop
+# because HeapTupleSatisfiesVacuum() would find the tuple HEAPTUPLE_DEAD
+# because its xmax preceded OldestXmin. This was fixed by passing OldestXmin
+# to heap_page_prune() and removing all tuples whose xmaxes precede
+# OldestXmin.
+#
+# With the fix, VACUUM should finish successfully, incrementing the table
+# vacuum_count.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT vacuum_count > 0
+ FROM pg_stat_all_tables WHERE relname = '${table1}';
+ ]
+ , 't');
+
+$primary_lsn = $node_primary->lsn('flush');
+
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);");
+
+# Nothing on the replica should cause a recovery conflict, so this should
+# finish successfully.
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+## Shut down psqls
+$psql_primaryA->quit;
+$psql_primaryB->quit;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
REL16_v2-0002-Ensure-vacuum-removes-all-visibly-dead-tuples-old.patch (text/x-patch, 6.5K) download
From 4bf49026066017399d5e60d8c4596be61bab580e Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 8 Jul 2024 16:54:35 -0400
Subject: [PATCH v2 2/2] Ensure vacuum removes all visibly dead tuples older
than OldestXmin
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than GlobalVisState->maybe_needed,
it will loop infinitely in lazy_scan_prune(), which compares tuples'
visibility information to OldestXmin.
Starting in version 14, which uses GlobalVisState for visibility testing
during pruning, it is possible for GlobalVisState->maybe_needed to
precede OldestXmin if maybe_needed is forced to go backward while vacuum
is running. This can happen if a disconnected standby with a running
transaction older than VacuumCutoffs->OldestXmin reconnects to the
primary after vacuum initially calculates GlobalVisState and OldestXmin.
Fix this by having vacuum always remove tuples older than OldestXmin
during pruning. This is okay because the standby won't replay the tuple
removal until the tuple is removable. Thus, the worst that can happen is
a recovery conflict.
Fixes BUG# 17257
Back-patched in versions 14-17
Author: Melanie Plageman
Reviewed-by: Noah Misch
Discussion: https://postgre.es/m/CAAKRu_Y_NJzF4-8gzTTeaOuUL3CcGoXPjXcAHbTTygT8AyVqag%40mail.gmail.com
---
src/backend/access/heap/pruneheap.c | 48 +++++++++++++++++++++-------
src/backend/access/heap/vacuumlazy.c | 2 +-
src/include/access/heapam.h | 1 +
3 files changed, 38 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 47b9e209154..f0c169e21f8 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -33,7 +33,8 @@ typedef struct
{
Relation rel;
- /* tuple visibility test, initialized for the relation */
+ /* State used to test tuple visibility; Initialized for the relation */
+ TransactionId oldest_xmin;
GlobalVisState *vistest;
/*
@@ -205,7 +206,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
int ndeleted,
nnewlpdead;
- ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
+ ndeleted = heap_page_prune(relation, buffer, InvalidTransactionId,
+ vistest, limited_xmin,
limited_ts, &nnewlpdead, NULL);
/*
@@ -247,11 +249,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* also need to account for a reduction in the length of the line pointer
* array following array truncation by us.
*
- * vistest is used to distinguish whether tuples are DEAD or RECENTLY_DEAD
- * (see heap_prune_satisfies_vacuum and
- * HeapTupleSatisfiesVacuum). old_snap_xmin / old_snap_ts need to
- * either have been set by TransactionIdLimitedForOldSnapshots, or
- * InvalidTransactionId/0 respectively.
+ * vistest and oldest_xmin are used to distinguish whether tuples are DEAD or
+ * RECENTLY_DEAD (see heap_prune_satisfies_vacuum and
+ * HeapTupleSatisfiesVacuum). If oldest_xmin is provided by the caller, it is
+ * used before consulting GlobalVisState.
+ *
+ * old_snap_xmin / old_snap_ts need to either have been set by
+ * TransactionIdLimitedForOldSnapshots, or InvalidTransactionId/0 respectively.
*
* Sets *nnewlpdead for caller, indicating the number of items that were
* newly set LP_DEAD during prune operation.
@@ -263,6 +267,7 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
int
heap_page_prune(Relation relation, Buffer buffer,
+ TransactionId oldest_xmin,
GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
@@ -290,6 +295,7 @@ heap_page_prune(Relation relation, Buffer buffer,
*/
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
+ prstate.oldest_xmin = oldest_xmin;
prstate.vistest = vistest;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
@@ -520,13 +526,31 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
}
/*
- * First check if GlobalVisTestIsRemovableXid() is sufficient to find the
- * row dead. If not, and old_snapshot_threshold is enabled, try to use the
- * lowered horizon.
+ * For VACUUM, we must be sure to prune tuples with xmax older than
+ * oldest_xmin -- a visibility cutoff determined at the beginning of
+ * vacuuming the relation. oldest_xmin is used for freezing determination
+ * and we cannot freeze dead tuples' xmaxes.
+ */
+ if (TransactionIdIsValid(prstate->oldest_xmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->oldest_xmin))
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * Determine whether or not the tuple is considered dead when compared
+ * with the provided GlobalVisState. On-access pruning does not provide
+ * oldest_xmin. And for vacuum, even if the tuple's xmax is not older than
+ * oldest_xmin, GlobalVisTestIsRemovableXid() could find the row dead if
+ * the GlobalVisState has been updated since the beginning of vacuuming
+ * the relation.
*/
if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
- res = HEAPTUPLE_DEAD;
- else if (OldSnapshotThresholdActive())
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * If GlobalVisTestIsRemovableXid() is not sufficient to find the row dead
+ * and old_snapshot_threshold is enabled, try to use the lowered horizon.
+ */
+ if (OldSnapshotThresholdActive())
{
/* haven't determined limited horizon yet, requests */
if (!TransactionIdIsValid(prstate->old_snap_xmin))
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index c474b061146..bd58ca8aae0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1581,7 +1581,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->cutoffs.OldestXmin, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index faf50265191..c7278219b24 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -285,6 +285,7 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
extern int heap_page_prune(Relation relation, Buffer buffer,
+ TransactionId oldest_xmin,
struct GlobalVisState *vistest,
TransactionId old_snap_xmin,
TimestampTz old_snap_ts,
--
2.34.1
REL17_v2-0001-Test-that-vacuum-removes-tuples-older-than-Oldest.patch (text/x-patch, 10.0K) download
From c6ebfaf3a967d97af3aa575df8c4190fad2117ee Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 20 Jun 2024 16:38:30 -0400
Subject: [PATCH v2 1/2] Test that vacuum removes tuples older than OldestXmin
If vacuum fails to prune a tuple killed before OldestXmin, it will
decide to freeze its xmax and later error out in pre-freeze checks.
---
src/test/recovery/meson.build | 1 +
.../recovery/t/043_vacuum_horizon_floor.pl | 236 ++++++++++++++++++
2 files changed, 237 insertions(+)
create mode 100644 src/test/recovery/t/043_vacuum_horizon_floor.pl
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec1..1d55d6bf560 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/043_vacuum_horizon_floor.pl',
],
},
}
diff --git a/src/test/recovery/t/043_vacuum_horizon_floor.pl b/src/test/recovery/t/043_vacuum_horizon_floor.pl
new file mode 100644
index 00000000000..1b82d935be5
--- /dev/null
+++ b/src/test/recovery/t/043_vacuum_horizon_floor.pl
@@ -0,0 +1,236 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# Test that vacuum prunes away all dead tuples killed before OldestXmin
+
+# Set up nodes
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 'physical');
+
+$node_primary->append_conf(
+ 'postgresql.conf', qq[
+hot_standby_feedback = on
+log_recovery_conflict_waits = true
+autovacuum = off
+maintenance_work_mem = 1024
+]);
+$node_primary->start;
+
+my $node_replica = PostgreSQL::Test::Cluster->new('standby');
+
+$node_primary->backup('my_backup');
+$node_replica->init_from_backup($node_primary, 'my_backup',
+ has_streaming => 1);
+
+$node_replica->start;
+
+my $test_db = "test_db";
+$node_primary->safe_psql('postgres', "CREATE DATABASE $test_db");
+
+# Save the original connection info for later use
+my $orig_conninfo = $node_primary->connstr();
+
+my $table1 = "vac_horizon_floor_table";
+
+# Long-running Primary Session A
+my $psql_primaryA =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Long-running Primary Session B
+my $psql_primaryB =
+ $node_primary->background_psql($test_db, on_error_stop => 1);
+
+# Insert and update enough rows that we force at least one round of index
+# vacuuming before getting to a dead tuple which was killed after the standby
+# is disconnected.
+#
+# We need multiple index vacuuming passes to repro because after the standby
+# reconnects to the primary, our backend's GlobalVisStates will not have been
+# updated with the new horizon until something forces them to be updated.
+#
+# _bt_pendingfsm_finalize() calls GetOldestNonRemovableTransactionId() at the
+# end of a round of index vacuuming, updating the backend's GlobalVisState
+# and, in our case, moving maybe_needed backwards.
+#
+# Then vacuum's first pass will continue and pruning will find our later
+# inserted and updated tuple HEAPTUPLE_RECENTLY_DEAD when compared to
+# maybe_needed but HEAPTUPLE_DEAD when compared to OldestXmin.
+$node_primary->safe_psql($test_db, qq[
+ CREATE TABLE ${table1}(col1 int) with (autovacuum_enabled=false, fillfactor=10);
+ INSERT INTO $table1 VALUES(7);
+ INSERT INTO $table1 SELECT generate_series(1, 400000) % 3;
+ CREATE INDEX on ${table1}(col1);
+ UPDATE $table1 SET col1 = 3 WHERE col1 = 0;
+ INSERT INTO $table1 VALUES(7);
+]);
+
+# We will later move the primary forward while the standby is disconnected. For
+# now, however, there is no reason not to wait for the standby to catch up.
+my $primary_lsn = $node_primary->lsn('flush');
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+# Test that the WAL receiver is up and running.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Set primary_conninfo to something invalid on the replica and reload the
+# config. Once the config is reloaded, the startup process will force the WAL
+# receiver to restart and it will be unable to reconnect because of the invalid
+# connection information.
+$node_replica->safe_psql($test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '';
+ SELECT pg_reload_conf();
+ ]);
+
+# Wait until the WAL receiver has shut down and been unable to start up again.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Now insert and update a tuple which will be visible to the vacuum on the
+# primary but which will have xmax newer than the oldest xmin on the standby
+# that was recently disconnected.
+my $res = $psql_primaryA->query_safe(
+ qq[
+ INSERT INTO $table1 VALUES (99);
+ UPDATE $table1 SET col1 = 100 WHERE col1 = 99;
+ SELECT 'after_update';
+ ]
+ );
+
+# Make sure the UPDATE finished
+like($res, qr/^after_update$/m, "UPDATE occurred on primary session A");
+
+# Open a cursor on the primary whose pin will keep VACUUM from getting a
+# cleanup lock on the first page of the relation. We want VACUUM to be able to
+# start, calculate initial values for OldestXmin and GlobalVisState and then be
+# unable to proceed with pruning our dead tuples. This will allow us to
+# reconnect the standby and push the horizon back before we start actual
+# pruning and vacuuming.
+my $primary_cursor1 = "vac_horizon_floor_cursor1";
+
+# The first value inserted into the table was a 7, so FETCH FORWARD should
+# return a 7. That's how we know the cursor has a pin.
+$res = $psql_primaryB->query_safe(
+ qq[
+ BEGIN;
+ DECLARE $primary_cursor1 CURSOR FOR SELECT col1 FROM $table1 WHERE col1 = 7;
+ FETCH $primary_cursor1;
+ ]
+ );
+
+is($res, 7, qq[Cursor query returned $res. Expected value 7.]);
+
+
+# Get the PID of the session which will run the VACUUM FREEZE so that we can
+# use it to filter pg_stat_activity later.
+my $vacuum_pid = $psql_primaryA->query_safe("SELECT pg_backend_pid();");
+
+# Now start a VACUUM FREEZE on the primary. It will call vacuum_get_cutoffs()
+# and establish values of OldestXmin and GlobalVisState which are newer than
+# all of our dead tuples. Then it will be unable to get a cleanup lock to start
+# pruning, so it will hang.
+$psql_primaryA->{stdin} .= qq[
+ VACUUM FREEZE $table1;
+ \\echo VACUUM
+ ];
+
+# Make sure the VACUUM command makes it to the server.
+$psql_primaryA->{run}->pump_nb();
+
+# Make sure that the VACUUM has already called vacuum_get_cutoffs() and is just
+# waiting on the lock to start vacuuming. We don't want the standby to
+# re-establish a connection to the primary and push the horizon back until
+# we've saved initial values in GlobalVisState and calculated OldestXmin.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT count(*) >= 1 FROM pg_stat_activity
+ WHERE pid = $vacuum_pid
+ AND wait_event = 'BufferPin';
+ ],
+ 't');
+
+# Ensure the WAL receiver is still not active on the replica.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 'f');
+
+# Allow the WAL receiver connection to re-establish.
+$node_replica->safe_psql(
+ $test_db, qq[
+ ALTER SYSTEM SET primary_conninfo = '$orig_conninfo';
+ SELECT pg_reload_conf();
+ ]);
+
+# Ensure the new WAL receiver has connected.
+$node_replica->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_wal_receiver);] , 't');
+
+# Once the WAL sender is shown on the primary, the replica should have
+# connected with the primary and pushed the horizon backward. Primary Session A
+# won't see that until the VACUUM FREEZE proceeds and does its first round of
+# index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ select exists (select * from pg_stat_replication);] , 't');
+
+# Move the cursor forward to the next 7. We inserted the 7 much later, so
+# advancing the cursor should allow vacuum to proceed vacuuming most pages of
+# the relation. Because we set maintanence_work_mem sufficiently low, we expect
+# that a round of index vacuuming has happened and that the vacuum is now
+# waiting for the cursor to release its pin on the last page of the relation.
+$res = $psql_primaryB->query_safe("FETCH $primary_cursor1");
+is($res, 7, qq[Cursor query returned $res from second fetch. Expected value 7.]);
+
+# Prevent the test from incorrectly passing by confirming that we did indeed do
+# a pass of index vacuuming.
+$node_primary->poll_query_until($test_db, qq[
+ SELECT index_vacuum_count > 0
+ FROM pg_stat_progress_vacuum
+ WHERE datname='$test_db' AND relid::regclass = '$table1'::regclass;
+ ] , 't');
+
+# Commit the cursor so that the VACUUM can finish.
+$psql_primaryB->query_until(
+ qr/^commit$/m,
+ qq[
+ COMMIT;
+ \\echo commit
+ ]
+ );
+
+# VACUUM proceeds with pruning and does a visibility check on each tuple. In
+# older versions of Postgres, pruning found our final dead tuple
+# non-removable (HEAPTUPLE_RECENTLY_DEAD) since its xmax is after the new
+# value of maybe_needed. Then heap_prepare_freeze_tuple() would decide the
+# tuple xmax should be frozen because it precedes OldestXmin. Vacuum would
+# then error out in heap_pre_freeze_checks() with "cannot freeze committed
+# xmax". This was fixed by changing pruning to find all
+# HEAPTUPLE_RECENTLY_DEAD tuples with xmaxes preceding OldestXmin
+# HEAPTUPLE_DEAD and removing them.
+
+# With the fix, VACUUM should finish successfully, incrementing the table
+# vacuum_count.
+$node_primary->poll_query_until($test_db,
+ qq[
+ SELECT vacuum_count > 0
+ FROM pg_stat_all_tables WHERE relname = '${table1}';
+ ]
+ , 't');
+
+$primary_lsn = $node_primary->lsn('flush');
+
+# Make sure something causes us to flush
+$node_primary->safe_psql($test_db, "INSERT INTO $table1 VALUES (1);");
+
+# Nothing on the replica should cause a recovery conflict, so this should
+# finish successfully.
+$node_primary->wait_for_catchup($node_replica, 'replay', $primary_lsn);
+
+## Shut down psqls
+$psql_primaryA->quit;
+$psql_primaryB->quit;
+
+$node_replica->stop();
+$node_primary->stop();
+
+done_testing();
--
2.34.1
REL17_v2-0002-Ensure-vacuum-removes-all-visibly-dead-tuples-old.patch (text/x-patch, 5.5K) download
From d7d5d61789731ca9d7839392049ad54a7eaddae7 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 20 Jun 2024 17:21:11 -0400
Subject: [PATCH v2 2/2] Ensure vacuum removes all visibly dead tuples older
than OldestXmin
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than GlobalVisState->maybe_needed,
it may attempt to freeze the tuple's xmax and then ERROR out in
pre-freeze checks with "cannot freeze committed xmax".
Fix this by having vacuum always remove tuples older than OldestXmin.
It is possible for GlobalVisState->maybe_needed to precede OldestXmin if
maybe_needed is forced to go backward while vacuum is running. This can
happen if a disconnected standby with a running transaction older than
VacuumCutoffs->OldestXmin reconnects to the primary after vacuum
initially calculates GlobalVisState and OldestXmin.
In back branches starting with 14, the first version using
GlobalVisState, failing to remove tuples older than OldestXmin during
pruning caused vacuum to infinitely loop in lazy_scan_prune(), as
investigated on this [1] thread. After 1ccc1e05ae removed the retry loop
in lazy_scan_prune() and stopped comparing tuples to OldestXmin, the
hang could no longer happen, but we could still attempt to freeze dead
tuples with xmax older than OldestXmin -- resulting in an ERROR.
Fix this by always removing dead tuples with xmax older than
VacuumCutoffs->OldestXmin. This is okay because the standby won't replay
the tuple removal until the tuple is removable. Thus, the worst that can
happen is a recovery conflict.
[1] https://postgr.es/m/20240415173913.4zyyrwaftujxthf2%40awork3.anarazel.de#1b216b7768b5bd577a3d3d51bd5aadee
Back-patch through 14
Author: Melanie Plageman
Reviewed-by: Peter Geoghegan, Robert Haas, Andres Freund, Heikki Linnakangas, and Noah Misch
Discussion: https://postgr.es/m/CAAKRu_bDD7oq9ZwB2OJqub5BovMG6UjEYsoK2LVttadjEqyRGg%40mail.gmail.com
---
src/backend/access/heap/pruneheap.c | 23 ++++++++++++++++++++++-
src/backend/access/heap/vacuumlazy.c | 14 +++++++-------
2 files changed, 29 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 3cdfc5b7f1b..869d82ad667 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -325,6 +325,8 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*
* cutoffs contains the freeze cutoffs, established by VACUUM at the beginning
* of vacuuming the relation. Required if HEAP_PRUNE_FREEZE option is set.
+ * cutoffs->OldestXmin is also used to determine if dead tuples are
+ * HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD.
*
* presult contains output parameters needed by callers, such as the number of
* tuples removed and the offsets of dead items on the page after pruning.
@@ -922,8 +924,27 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
if (res != HEAPTUPLE_RECENTLY_DEAD)
return res;
+ /*
+ * For VACUUM, we must be sure to prune tuples with xmax older than
+ * OldestXmin -- a visibility cutoff determined at the beginning of
+ * vacuuming the relation. OldestXmin is used for freezing determination
+ * and we cannot freeze dead tuples' xmaxes.
+ */
+ if (prstate->cutoffs &&
+ TransactionIdIsValid(prstate->cutoffs->OldestXmin) &&
+ NormalTransactionIdPrecedes(dead_after, prstate->cutoffs->OldestXmin))
+ return HEAPTUPLE_DEAD;
+
+ /*
+ * Determine whether or not the tuple is considered dead when compared
+ * with the provided GlobalVisState. On-access pruning does not provide
+ * VacuumCutoffs. And for vacuum, even if the tuple's xmax is not older
+ * than OldestXmin, GlobalVisTestIsRemovableXid() could find the row dead
+ * if the GlobalVisState has been updated since the beginning of vacuuming
+ * the relation.
+ */
if (GlobalVisTestIsRemovableXid(prstate->vistest, dead_after))
- res = HEAPTUPLE_DEAD;
+ return HEAPTUPLE_DEAD;
return res;
}
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 3f88cf1e8ef..abb47ae5960 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -438,13 +438,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in pruning. We
- * expect vistest will always make heap_page_prune_and_freeze() remove any
- * deleted tuple whose xmax is < OldestXmin. lazy_scan_prune must never
- * become confused about whether a tuple should be frozen or removed. (In
- * the future we might want to teach lazy_scan_prune to recompute vistest
- * from time to time, to increase the number of dead tuples it can prune
- * away.)
+ * Next acquire vistest, a related cutoff that's used in pruning. We use
+ * vistest in combination with OldestXmin to ensure that
+ * heap_page_prune_and_freeze() always removes any deleted tuple whose
+ * xmax is < OldestXmin. lazy_scan_prune must never become confused about
+ * whether a tuple should be frozen or removed. (In the future we might
+ * want to teach lazy_scan_prune to recompute vistest from time to time,
+ * to increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
--
2.34.1
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-07-11 21:16 Noah Misch <noah@leadboat.com>
parent: Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 62+ messages in thread
From: Noah Misch @ 2024-07-11 21:16 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Bowen Shi <zxwsbg12138@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Jul 08, 2024 at 05:23:52PM -0400, Melanie Plageman wrote:
> I am nervous about the test flaking on the buildfarm. But I did
> the best I could to try to make it stable. I think keeping it as a
> separate commit should be easiest in case we have to revert it?
Whether one uses a separate commit or not, it's easy to remove a new test. I
think either way is fine. We've not typically committed tests separately.
^ permalink raw reply [nested|flat] 62+ messages in thread
* Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae
@ 2024-07-12 01:20 Michael Paquier <michael@paquier.xyz>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 62+ messages in thread
From: Michael Paquier @ 2024-07-12 01:20 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Bowen Shi <zxwsbg12138@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Jul 11, 2024 at 02:16:08PM -0700, Noah Misch wrote:
> On Mon, Jul 08, 2024 at 05:23:52PM -0400, Melanie Plageman wrote:
>> I am nervous about the test flaking on the buildfarm. But I did
>> the best I could to try to make it stable. I think keeping it as a
>> separate commit should be easiest in case we have to revert it?
>
> Whether one uses a separate commit or not, it's easy to remove a new test. I
> think either way is fine. We've not typically committed tests separately.
Depends. Melanie, if you are nervous about the test stability, I'd
suggest to do what you think is better and separate that into two
commits to ease your flow, then. This will limit the damage should a
revert of the test be done. I'm pretty sure that I have done that in
the past more than once when I had similar concerns in mind.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 62+ messages in thread
end of thread, other threads:[~2024-07-12 01:20 UTC | newest]
Thread overview: 62+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-03-04 00:07 relfrozenxid may disagree with row XIDs after 1ccc1e05ae Noah Misch <noah@leadboat.com>
2024-03-21 16:15 ` Robert Haas <robertmhaas@gmail.com>
2024-03-21 17:21 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2024-03-22 12:22 ` Robert Haas <robertmhaas@gmail.com>
2024-03-22 18:41 ` Melanie Plageman <melanieplageman@gmail.com>
2024-03-22 19:34 ` Robert Haas <robertmhaas@gmail.com>
2024-03-29 16:05 ` Robert Haas <robertmhaas@gmail.com>
2024-03-29 17:16 ` Peter Geoghegan <pg@bowt.ie>
2024-03-29 18:27 ` Robert Haas <robertmhaas@gmail.com>
2024-03-30 16:19 ` Melanie Plageman <melanieplageman@gmail.com>
2024-03-30 17:35 ` Robert Haas <robertmhaas@gmail.com>
2024-04-15 16:35 ` Robert Haas <robertmhaas@gmail.com>
2024-04-15 18:04 ` Andres Freund <andres@anarazel.de>
2024-04-15 20:52 ` Noah Misch <noah@leadboat.com>
2024-04-15 21:10 ` Andres Freund <andres@anarazel.de>
2024-04-16 03:58 ` Noah Misch <noah@leadboat.com>
2024-04-16 18:01 ` Andres Freund <andres@anarazel.de>
2024-04-16 19:34 ` Noah Misch <noah@leadboat.com>
2024-04-15 23:08 ` Michael Paquier <michael@paquier.xyz>
2024-05-13 03:19 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-13 14:42 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-14 03:42 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-14 23:33 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-15 02:43 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-15 13:26 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-16 02:21 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-16 03:33 ` Andres Freund <andres@anarazel.de>
2024-05-16 07:31 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-16 16:38 ` Andres Freund <andres@anarazel.de>
2024-05-16 16:49 ` Peter Geoghegan <pg@bowt.ie>
2024-05-16 19:39 ` Andres Freund <andres@anarazel.de>
2024-05-16 20:03 ` Andres Freund <andres@anarazel.de>
2024-05-16 20:13 ` Peter Geoghegan <pg@bowt.ie>
2024-05-16 20:29 ` Andres Freund <andres@anarazel.de>
2024-05-16 20:37 ` Peter Geoghegan <pg@bowt.ie>
2024-05-16 20:38 ` Andres Freund <andres@anarazel.de>
2024-05-17 03:35 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-17 04:28 ` Peter Geoghegan <pg@bowt.ie>
2024-05-28 06:31 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-05-28 06:38 ` Peter Geoghegan <pg@bowt.ie>
2024-05-17 02:12 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-04-15 18:52 ` Andres Freund <andres@anarazel.de>
2024-04-26 20:46 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-09 21:56 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-16 23:57 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-18 22:23 ` Noah Misch <noah@leadboat.com>
2024-05-20 15:58 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-20 20:48 ` Noah Misch <noah@leadboat.com>
2024-05-22 16:57 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-22 23:59 ` Noah Misch <noah@leadboat.com>
2024-05-28 08:29 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-06-17 21:29 ` Melanie Plageman <melanieplageman@gmail.com>
2024-05-28 09:03 ` Bowen Shi <zxwsbg12138@gmail.com>
2024-06-18 22:39 ` Melanie Plageman <melanieplageman@gmail.com>
2024-06-18 22:51 ` Melanie Plageman <melanieplageman@gmail.com>
2024-06-20 15:49 ` Melanie Plageman <melanieplageman@gmail.com>
2024-06-21 00:33 ` Melanie Plageman <melanieplageman@gmail.com>
2024-06-25 19:37 ` Noah Misch <noah@leadboat.com>
2024-07-08 21:23 ` Melanie Plageman <melanieplageman@gmail.com>
2024-07-11 21:16 ` Noah Misch <noah@leadboat.com>
2024-07-12 01:20 ` Michael Paquier <michael@paquier.xyz>
2024-03-22 19:43 ` Noah Misch <noah@leadboat.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox