MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Thu, 29 Jun 2023 18:42:37 +0900
From: torikoshia <torikoshia@oss.nttdata.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: bungina@gmail.com, cyberdemn@gmail.com,
 pgsql-hackers@lists.postgresql.org
Subject: Re: pg_rewind WAL segments deletion pitfall
In-Reply-To: <20230629.102533.2256222097295418108.horikyota.ntt@gmail.com>
References: <20220928.181739.1943880428843903268.horikyota.ntt@gmail.com>
 <CAAtGL4CM7hA0G4dS-Z_RckMy+pzz_KGV2b=8ivDPTgaU4nySGg@mail.gmail.com>
 <2e75ae22dce9a227c3d47fa6d0ed094a@oss.nttdata.com>
 <20230629.102533.2256222097295418108.horikyota.ntt@gmail.com>
User-Agent: Roundcube Webmail/1.4.11
Message-ID: <a06c11b58c462f314a061ad64b3f6353@oss.nttdata.com>
Archived-At: 
 <https://www.postgresql.org/message-id/a06c11b58c462f314a061ad64b3f6353%40oss.nttdata.com>
Precedence: bulk

On 2023-06-29 10:25, Kyotaro Horiguchi wrote:
Thanks for the comment!

> At Wed, 28 Jun 2023 22:28:13 +0900, torikoshia
> <torikoshia@oss.nttdata.com> wrote in
>> 
>> On 2022-09-29 17:18, Polina Bungina wrote:
>> > I agree with your suggestions, so here is the updated version of
>> > patch. Hope I haven't missed anything.
>> > Regards,
>> > Polina Bungina
>> 
>> Thanks for working on this!
>> It seems like we are also facing the same issue.
> 
> Thanks for looking this.
> 
>> I tested the v3 patch under our condition, old primary has succeeded
>> to become new standby.
>> 
>> 
>> BTW when I used pg_rewind-removes-wal-segments-reproduce.sh attached
>> in [1], old primary also failed to become standby:
>> 
>>   FATAL: could not receive data from WAL stream: ERROR: requested WAL
>>   segment 000000020000000000000007 has already been removed
>> 
>> However, I think this is not a problem: just adding restore_command
>> like below fixed the situation.
>> 
>>   echo "restore_command = '/bin/cp `pwd`/newarch/%f %p'" >>
>>   oldprim/postgresql.conf
> 
> I thought on the same line at first, but that's not the point
> here.

Yes. I don't think adding restore_command solves the problem and
modification to prevent deleting necessary WAL like proposed
patch is necessary.

I added restore_command since
pg_rewind-removes-wal-segments-reproduce.sh failed to catch up
even after applying v3 patch and prevent pg_rewind from delete
WALs(*), because some necessary WALs were archived.

It's not a problem we are discussing here, but I wanted to get
the script to work to the point where old primary could
successfully catch up to new primary.

(*)Specifically, running the script without apply the patch,
recovery failed because 000000010000000000000003 which has
already been removed. This file was deleted by pg_rewind as
we know.
OTHO without the restore_command, recovery failed because
000000020000000000000007 has already been removed even after
applying the patch.

> The problem we want ot address is that pg_rewind ultimately
> removes certain crucial WAL files required for the new primary to
> start, despite them being present previously.

I thought it's not "new primary", but "old primary".

> In other words, that
> restore_command works, but it only undoes what pg_rewind wrongly did,
> resulting in unnecessary consupmtion of I/O and/or network bandwidth
> that essentially serves no purpose.

As far as I tested using the script and the situation we are facing,
after promoting newprim necessary WAL(000000010000000000000003..) were
not available and just adding restore_command did not solve the problem.

> pg_rewind already has a feature that determines how each file should
> be handled, but it is currently making wrong dicisions for WAL
> files. The goal here is to rectify this behavior and ensure that
> pg_rewind makes the right decisions.

+1


-- 
Regards,

--
Atsushi Torikoshi
NTT DATA CORPORATION