public inbox for [email protected]
help / color / mirror / Atom feedFrom: Alexander Korotkov <[email protected]>
To: Hayato Kuroda (Fujitsu) <[email protected]>
Cc: Amit Kapila <[email protected]>
Cc: Vitaly Davydov <[email protected]>
Cc: vignesh C <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Alexander Lakhin <[email protected]>
Cc: pgsql-hackers <[email protected]>
Cc: Masahiko Sawada <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly
Date: Fri, 20 Jun 2025 03:24:16 +0300
Message-ID: <CAPpHfdvjGWo--xqqjJbyb_amdkhqamnzrwCZWe_hBD-rSTFbBg@mail.gmail.com> (raw)
In-Reply-To: <OSCPR01MB1496646E4CF72B9231DB711AFF57DA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
References: <1d12d2-67235980-35-19a406a0@63439497>
<CAPpHfdtp61pSt72KxxOnnV_Oj4OTAiKYCSTek0Pm87hVDod4Bg@mail.gmail.com>
<CAA4eK1Lmd-stXWrNqX9iXugjY=ahhe7zRfyq+-vN512xvF4G2w@mail.gmail.com>
<CAPpHfdsTiYQPywr4yeqVhUtA+eFdqisqcAOiTb+kiYhi9EVEvw@mail.gmail.com>
<CAPpHfdvk-nfu5xzJyPArxWctofWPR+1L9uJ+yyWFA34n6b5rGQ@mail.gmail.com>
<CAA4eK1Ladnm40UO-YC+c7eBYqjyFQSwg0dn1w_CiaMoBTsJUDA@mail.gmail.com>
<CAPpHfduX--KKX853UwaJ8Cjo5bxbh19V_+McKQ3p9aS6b8T1Wg@mail.gmail.com>
<CAA4eK1+VbBxYY-rG4B7Jq6TsYvdjt3d2nbUrs7p9-1sP5mH+sw@mail.gmail.com>
<CAPpHfdvxqjE+RZdoOODQZAmQip6NpvSH40DuvVFWuf_00ek-yQ@mail.gmail.com>
<1d97ec-6841b600-3-b7b4780@40747246>
<CAA4eK1LvssAcp61qg957xRF0=K4xF0hfJQjQ6c9V7EGHSqraPg@mail.gmail.com>
<28c8bf-68470780-3-51b29480@89454035>
<CAPpHfdt77k3BqD=anh4b7UiRevfPOu2Hcf3rvSZXntxH+4G=Ug@mail.gmail.com>
<[email protected]>
<CAPpHfdvPpg5VUTNOqfKSNx66ij3ZJNZbf4kn6SDcPkdq9FTvng@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAPpHfdvJMe4i0_wp1tq2ZGSDKqUHKPwkfD_Y6KyJ3XQRFvc0cQ@mail.gmail.com>
<OSCPR01MB1496667BEA93302F8903EE512F570A@OSCPR01MB14966.jpnprd01.prod.outlook.com>
<CAPpHfdsdJ4YPTzH=cnany5_LjCNb_Z9e+Oucv53qYQHaHzJhag@mail.gmail.com>
<CALDaNm3s-jpQTe1MshsvQ8GO=TLj233JCdkQ7uZ6pwqRVpxAdw@mail.gmail.com>
<3f242e-68518380-11-78125f00@134027639>
<CAPpHfds0ScFbExsT+pnY7K0JGH6pFPHMOSfCOknC-1s7ahmRFA@mail.gmail.com>
<214ba-6852df80-9-7e0df800@134698799>
<21b44-6852e000-3-a556820@87831171>
<CAPpHfdv3UEUBjsLhB_CwJT0xX9LmN6U+__myYopq4KcgvCSbTg@mail.gmail.com>
<CAA4eK1+qL9QFOD2Q5kcq0Ff=7OcBLE34QuDBdPcdznUzQwv+eg@mail.gmail.com>
<OSCPR01MB1496646E4CF72B9231DB711AFF57DA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
Dear Kuroda-san,
On Thu, Jun 19, 2025 at 2:05 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
> > > Regarding assertion failure, I've found that assert in
> > > PhysicalConfirmReceivedLocation() conflicts with restart_lsn
> > > previously set by ReplicationSlotReserveWal(). As I can see,
> > > ReplicationSlotReserveWal() just picks fresh XLogCtl->RedoRecPtr lsn.
> > > So, it doesn't seems there is a guarantee that restart_lsn never goes
> > > backward. The commit in ReplicationSlotReserveWal() even states there
> > > is a "chance that we have to retry".
> > >
> >
> > I don't see how this theory can lead to a restart_lsn of a slot going
> > backwards. The retry mentioned there is just a retry to reserve the
> > slot's position again if the required WAL is already removed. Such a
> > retry can only get the position later than the previous restart_lsn.
>
> We analyzed the assertion failure happened at pg_basebackup/020_pg_receivewal,
> and confirmed that restart_lsn can go backward. This meant that Assert() added
> by the ca307d5 can cause crash.
>
> Background
> ===========
> When pg_receivewal starts the replication and it uses the replication slot, it
> sets as the beginning of the segment where restart_lsn exists, as the startpoint.
> E.g., if the restart_lsn of the slot is 0/B000D0, pg_receivewal requests WALs
> from 0/B00000.
> More detail of this behavior, see f61e1dd2 and d9bae531.
>
> What happened here
> ==================
> Based on above theory, walsender sent from the beginning of segment (0/B00000).
> When walreceiver receives, it tried to send reply. At that time the flushed
> location of WAL would be 0/B00000. walsender sets the received lsn as restart_lsn
> in PhysicalConfirmReceivedLocation(). Here the restart_lsn went backward (0/B000D0->0/B00000).
>
> The assertion failure could happen if CHECKPOINT happened at that time.
> Attribute last_saved_restart_lsn of the slot was 0/B000D0, but the data.restart_lsn
> was 0/B00000. It could not satisfy the assertion added in InvalidatePossiblyObsoleteSlot().
Thank you for your detailed explanation!
> Note
> ====
> 1.
> In this case, starting from the beginning of the segment is not a problem, because
> the checkpoint process only removes WAL files from segments that precede the
> restart_lsn's wal segment. The current segment (0/B00000) will not be removed,
> so there is no risk of data loss or inconsistency.
>
> 2.
> A similar pattern applies to pg_basebackup. Both use logic that adjusts the
> requested streaming position to the start of the segment, and it replies the
> received LSN as flushed.
>
> 3.
> I considered the theory above, but I could not reproduce 040_standby_failover_slots_sync
> because it is a timing issue. Have someone else reproduced?
>
> We are still investigating failure caused at 040_standby_failover_slots_sync.
I didn't manage to reproduce this. But as I see from the logs [1] on
mamba that START_REPLICATION command was issued just before assert
trap. Could it be something similar to what I described in [2].
Namely:
1. ReplicationSlotReserveWal() sets restart_lsn for the slot.
2. Concurrent checkpoint flushes that restart_lsn to the disk.
3. PhysicalConfirmReceivedLocation() sets restart_lsn of the slot to
the beginning of the segment.
[1] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=mamba&dt=2025-06-17%2005%3A10%3A36...
[2] https://www.postgresql.org/message-id/CAPpHfdv3UEUBjsLhB_CwJT0xX9LmN6U%2B__myYopq4KcgvCSbTg%40mail.g...
------
Regards,
Alexander Korotkov
Supabase
view thread (73+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly
In-Reply-To: <CAPpHfdvjGWo--xqqjJbyb_amdkhqamnzrwCZWe_hBD-rSTFbBg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox