MIME-Version: 1.0
References: <1d12d2-67235980-35-19a406a0@63439497>
 <CAPpHfdtp61pSt72KxxOnnV_Oj4OTAiKYCSTek0Pm87hVDod4Bg@mail.gmail.com>
 <CAA4eK1Lmd-stXWrNqX9iXugjY=ahhe7zRfyq+-vN512xvF4G2w@mail.gmail.com>
 <CAPpHfdsTiYQPywr4yeqVhUtA+eFdqisqcAOiTb+kiYhi9EVEvw@mail.gmail.com>
 <CAPpHfdvk-nfu5xzJyPArxWctofWPR+1L9uJ+yyWFA34n6b5rGQ@mail.gmail.com>
 <CAA4eK1Ladnm40UO-YC+c7eBYqjyFQSwg0dn1w_CiaMoBTsJUDA@mail.gmail.com>
 <CAPpHfduX--KKX853UwaJ8Cjo5bxbh19V_+McKQ3p9aS6b8T1Wg@mail.gmail.com>
 <CAA4eK1+VbBxYY-rG4B7Jq6TsYvdjt3d2nbUrs7p9-1sP5mH+sw@mail.gmail.com>
 <CAPpHfdvxqjE+RZdoOODQZAmQip6NpvSH40DuvVFWuf_00ek-yQ@mail.gmail.com>
 <1d97ec-6841b600-3-b7b4780@40747246>
 <CAA4eK1LvssAcp61qg957xRF0=K4xF0hfJQjQ6c9V7EGHSqraPg@mail.gmail.com>
 <28c8bf-68470780-3-51b29480@89454035>
 <CAPpHfdt77k3BqD=anh4b7UiRevfPOu2Hcf3rvSZXntxH+4G=Ug@mail.gmail.com>
 <fbc5d94e-6fbd-4a64-85d4-c9e284a58eb2@gmail.com>
 <CAPpHfdvPpg5VUTNOqfKSNx66ij3ZJNZbf4kn6SDcPkdq9FTvng@mail.gmail.com>
 <0a617d34-abc3-4090-92a5-ea15d45e36f6@gmail.com>
 <228607.1750003556@sss.pgh.pa.us>
 <CAPpHfdvJMe4i0_wp1tq2ZGSDKqUHKPwkfD_Y6KyJ3XQRFvc0cQ@mail.gmail.com>
 <OSCPR01MB1496667BEA93302F8903EE512F570A@OSCPR01MB14966.jpnprd01.prod.outlook.com>
 <CAPpHfdsdJ4YPTzH=cnany5_LjCNb_Z9e+Oucv53qYQHaHzJhag@mail.gmail.com>
 <CALDaNm3s-jpQTe1MshsvQ8GO=TLj233JCdkQ7uZ6pwqRVpxAdw@mail.gmail.com>
 <3f242e-68518380-11-78125f00@134027639>
 <CAPpHfds0ScFbExsT+pnY7K0JGH6pFPHMOSfCOknC-1s7ahmRFA@mail.gmail.com>
 <214ba-6852df80-9-7e0df800@134698799> <21b44-6852e000-3-a556820@87831171>
 <CAPpHfdv3UEUBjsLhB_CwJT0xX9LmN6U+__myYopq4KcgvCSbTg@mail.gmail.com>
 <CAA4eK1+qL9QFOD2Q5kcq0Ff=7OcBLE34QuDBdPcdznUzQwv+eg@mail.gmail.com>
 <OSCPR01MB1496646E4CF72B9231DB711AFF57DA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
In-Reply-To: 
 <OSCPR01MB1496646E4CF72B9231DB711AFF57DA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
From: Alexander Korotkov <aekorotkov@gmail.com>
Date: Fri, 20 Jun 2025 03:24:16 +0300
Message-ID: 
 <CAPpHfdvjGWo--xqqjJbyb_amdkhqamnzrwCZWe_hBD-rSTFbBg@mail.gmail.com>
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard
 restart unexpectedly
To: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>,
 Vitaly Davydov <v.davydov@postgrespro.ru>,
	vignesh C <vignesh21@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>,
	Alexander Lakhin <exclusion@gmail.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>,
	Masahiko Sawada <sawada.mshk@gmail.com>, "tomas@vondra.me" <tomas@vondra.me>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAPpHfdvjGWo--xqqjJbyb_amdkhqamnzrwCZWe_hBD-rSTFbBg%40mail.gmail.com>
Precedence: bulk

Dear Kuroda-san,

On Thu, Jun 19, 2025 at 2:05=E2=80=AFPM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
> > > Regarding assertion failure, I've found that assert in
> > > PhysicalConfirmReceivedLocation() conflicts with restart_lsn
> > > previously set by ReplicationSlotReserveWal().  As I can see,
> > > ReplicationSlotReserveWal() just picks fresh XLogCtl->RedoRecPtr lsn.
> > > So, it doesn't seems there is a guarantee that restart_lsn never goes
> > > backward.  The commit in ReplicationSlotReserveWal() even states ther=
e
> > > is a "chance that we have to retry".
> > >
> >
> > I don't see how this theory can lead to a restart_lsn of a slot going
> > backwards. The retry mentioned there is just a retry to reserve the
> > slot's position again if the required WAL is already removed. Such a
> > retry can only get the position later than the previous restart_lsn.
>
> We analyzed the assertion failure happened at pg_basebackup/020_pg_receiv=
ewal,
> and confirmed that restart_lsn can go backward. This meant that Assert() =
added
> by the ca307d5 can cause crash.
>
> Background
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> When pg_receivewal starts the replication and it uses the replication slo=
t, it
> sets as the beginning of the segment where restart_lsn exists, as the sta=
rtpoint.
> E.g., if the restart_lsn of the slot is 0/B000D0, pg_receivewal requests =
WALs
> from 0/B00000.
> More detail of this behavior, see f61e1dd2 and d9bae531.
>
> What happened here
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Based on above theory, walsender sent from the beginning of segment (0/B0=
0000).
> When walreceiver receives, it tried to send reply. At that time the flush=
ed
> location of WAL would be 0/B00000. walsender sets the received lsn as res=
tart_lsn
> in PhysicalConfirmReceivedLocation(). Here the restart_lsn went backward =
(0/B000D0->0/B00000).
>
> The assertion failure could happen if CHECKPOINT happened at that time.
> Attribute last_saved_restart_lsn of the slot was 0/B000D0, but the data.r=
estart_lsn
> was 0/B00000. It could not satisfy the assertion added in InvalidatePossi=
blyObsoleteSlot().

Thank you for your detailed explanation!

> Note
> =3D=3D=3D=3D
> 1.
> In this case, starting from the beginning of the segment is not a problem=
, because
> the checkpoint process only removes WAL files from segments that precede =
the
> restart_lsn's wal segment. The current segment (0/B00000) will not be rem=
oved,
> so there is no risk of data loss or inconsistency.
>
> 2.
> A similar pattern applies to pg_basebackup. Both use logic that adjusts t=
he
> requested streaming position to the start of the segment, and it replies =
the
> received LSN as flushed.
>
> 3.
> I considered the theory above, but I could not reproduce 040_standby_fail=
over_slots_sync
> because it is a timing issue. Have someone else reproduced?
>
> We are still investigating failure caused at 040_standby_failover_slots_s=
ync.

I didn't manage to reproduce this.  But as I see from the logs [1] on
mamba that START_REPLICATION command was issued just before assert
trap.  Could it be something similar to what I described in [2].
Namely:
1. ReplicationSlotReserveWal() sets restart_lsn for the slot.
2. Concurrent checkpoint flushes that restart_lsn to the disk.
3. PhysicalConfirmReceivedLocation() sets restart_lsn of the slot to
the beginning of the segment.

[1] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=3Dmamba&d=
t=3D2025-06-17%2005%3A10%3A36&stg=3Drecovery-check
[2] https://www.postgresql.org/message-id/CAPpHfdv3UEUBjsLhB_CwJT0xX9LmN6U%=
2B__myYopq4KcgvCSbTg%40mail.gmail.com

------
Regards,
Alexander Korotkov
Supabase