agora inbox for pgsql-bugs@postgresql.org
help / color / mirror / Atom feedFrom: torikoshia <torikoshia@oss.nttdata.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: bungina@gmail.com
Cc: cyberdemn@gmail.com
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: pg_rewind WAL segments deletion pitfall
Date: Thu, 29 Jun 2023 18:42:37 +0900
Message-ID: <a06c11b58c462f314a061ad64b3f6353@oss.nttdata.com> (raw)
In-Reply-To: <20230629.102533.2256222097295418108.horikyota.ntt@gmail.com>
References: <20220928.181739.1943880428843903268.horikyota.ntt@gmail.com>
<CAAtGL4CM7hA0G4dS-Z_RckMy+pzz_KGV2b=8ivDPTgaU4nySGg@mail.gmail.com>
<2e75ae22dce9a227c3d47fa6d0ed094a@oss.nttdata.com>
<20230629.102533.2256222097295418108.horikyota.ntt@gmail.com>
On 2023-06-29 10:25, Kyotaro Horiguchi wrote:
Thanks for the comment!
> At Wed, 28 Jun 2023 22:28:13 +0900, torikoshia
> <torikoshia@oss.nttdata.com> wrote in
>>
>> On 2022-09-29 17:18, Polina Bungina wrote:
>> > I agree with your suggestions, so here is the updated version of
>> > patch. Hope I haven't missed anything.
>> > Regards,
>> > Polina Bungina
>>
>> Thanks for working on this!
>> It seems like we are also facing the same issue.
>
> Thanks for looking this.
>
>> I tested the v3 patch under our condition, old primary has succeeded
>> to become new standby.
>>
>>
>> BTW when I used pg_rewind-removes-wal-segments-reproduce.sh attached
>> in [1], old primary also failed to become standby:
>>
>> FATAL: could not receive data from WAL stream: ERROR: requested WAL
>> segment 000000020000000000000007 has already been removed
>>
>> However, I think this is not a problem: just adding restore_command
>> like below fixed the situation.
>>
>> echo "restore_command = '/bin/cp `pwd`/newarch/%f %p'" >>
>> oldprim/postgresql.conf
>
> I thought on the same line at first, but that's not the point
> here.
Yes. I don't think adding restore_command solves the problem and
modification to prevent deleting necessary WAL like proposed
patch is necessary.
I added restore_command since
pg_rewind-removes-wal-segments-reproduce.sh failed to catch up
even after applying v3 patch and prevent pg_rewind from delete
WALs(*), because some necessary WALs were archived.
It's not a problem we are discussing here, but I wanted to get
the script to work to the point where old primary could
successfully catch up to new primary.
(*)Specifically, running the script without apply the patch,
recovery failed because 000000010000000000000003 which has
already been removed. This file was deleted by pg_rewind as
we know.
OTHO without the restore_command, recovery failed because
000000020000000000000007 has already been removed even after
applying the patch.
> The problem we want ot address is that pg_rewind ultimately
> removes certain crucial WAL files required for the new primary to
> start, despite them being present previously.
I thought it's not "new primary", but "old primary".
> In other words, that
> restore_command works, but it only undoes what pg_rewind wrongly did,
> resulting in unnecessary consupmtion of I/O and/or network bandwidth
> that essentially serves no purpose.
As far as I tested using the script and the situation we are facing,
after promoting newprim necessary WAL(000000010000000000000003..) were
not available and just adding restore_command did not solve the problem.
> pg_rewind already has a feature that determines how each file should
> be handled, but it is currently making wrong dicisions for WAL
> files. The goal here is to rectify this behavior and ensure that
> pg_rewind makes the right decisions.
+1
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
view thread (50+ messages) latest in thread
Message-ID: <a06c11b58c462f314a061ad64b3f6353@oss.nttdata.com>
Permalink: ../a06c11b58c462f314a061ad64b3f6353@oss.nttdata.com/
Also on: postgresql.org/message-id/a06c11b58c462f314a061ad64b3f6353@oss.nttdata.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: torikoshia@oss.nttdata.com, horikyota.ntt@gmail.com, bungina@gmail.com, cyberdemn@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: pg_rewind WAL segments deletion pitfall
In-Reply-To: <a06c11b58c462f314a061ad64b3f6353@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox