Date: Wed, 7 Apr 2021 13:32:18 -0700
From: Andres Freund <andres@anarazel.de>
To: "Drouvot, Bertrand" <bdrouvot@amazon.com>
Cc: fabriziomello@gmail.com, Alvaro Herrera <alvherre@2ndquadrant.com>,
	tushar <tushar.ahuja@enterprisedb.com>,
	"[pgdg] Robert Haas" <robertmhaas@gmail.com>,
	Amit Khandekar <amitdkhan.pg@gmail.com>,
	Rahila Syed <rahila.syed@2ndquadrant.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Minimal logical decoding on standbys
Message-ID: <20210407203218.pestfxu7watrnog4@alap3.anarazel.de>
References: 
 <CAFcNs+rwMtfBkNkAodB9mZUAxyqvYHV0FzANm+uRiDLW6O7q1g@mail.gmail.com>
 <CAFcNs+q9FvSW7gMqM__aeWOoDNqfcQCoB8LiO-X33fvqGO8mkQ@mail.gmail.com>
 <96644a03-0b4d-e0a6-2124-0bc37e5b54fb@amazon.com>
 <CAFcNs+pyDR+iwYpTAp=cnkX7BgEbOnF6rrr8w=2inPAn4rQB-A@mail.gmail.com>
 <842d60ce-99e6-f35e-8c92-0cd49381326e@amazon.com>
 <CAFcNs+pQ0jBvUUH8XzE9ct64Doy-pJsfiT13StsaLpmviPAa8Q@mail.gmail.com>
 <e6cfd6c9-29de-b7dc-d008-446fdafe1f08@amazon.com>
 <6b9168e5-6a29-7fda-8903-cd50563fa84c@amazon.com>
 <20210406180231.qsnkyrgrm7gtxb73@alap3.anarazel.de>
 <20210407170954.xbovgh3xdrnogg7w@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210407170954.xbovgh3xdrnogg7w@alap3.anarazel.de>
Archived-At: 
 <https://www.postgresql.org/message-id/20210407203218.pestfxu7watrnog4%40alap3.anarazel.de>
Precedence: bulk

Hi,

On 2021-04-07 10:09:54 -0700, Andres Freund wrote:
> There's also no test for a recovery conflict due to row removal. Despite
> that being a substantial part of the patchset.

Another aspect that wasn't tested *at all*: Whether logical decoding
actually produces useful and correct results.


> I'm tempted to throw out 024 - all of its tests seem fragile and prove
> little. And then add a few more tests to 025 (and renaming it).

While working on this I found a, somewhat substantial, issue:

When the primary is idle, on the standby logical decoding via walsender
will typically not process the records until further WAL writes come in
from the primary, or until a 10s lapsed.

The problem is that WalSndWaitForWal() waits for the *replay* LSN to
increase, but gets woken up by walreceiver when new WAL has been
flushed. Which means that typically walsenders will get woken up at the
same time that the startup process will be - which means that by the
time the logical walsender checks GetXLogReplayRecPtr() it's unlikely
that the startup process already replayed the record and updated
XLogCtl->lastReplayedEndRecPtr.

I think fixing this would require too invasive changes at this point. I
think we might be able to live with 10s delay issue for one release, but
it sure is ugly :(.

Greetings,

Andres Freund