Subject: Re: speed up a logical replica setup

From: Peter Eisentraut <[email-masked]>
Date: Tue, 7 Nov 2023 22:00:39 +0100

On 23.10.23 05:53, Euler Taveira wrote:
> Let's continue with the bike shedding... I agree with Peter E that this name
> does not express what this tool is. At the moment, it only have one action:
> create. If I have to suggest other actions I would say that it could support
> switchover option too (that removes the infrastructure created by this 
> tool).
> If we decide to keep this name, it should be a good idea to add an option to
> indicate what action it is executing (similar to pg_recvlogical) as 
> suggested
> by Peter.

Speaking of which, would it make sense to put this tool (whatever the 
name) into the pg_basebackup directory?  It's sort of related, and it 
also shares some code.

---

From: Michael Paquier <[email-masked]>
Date: Wed, 8 Nov 2023 08:12:50 +0900

On Tue, Nov 07, 2023 at 10:00:39PM +0100, Peter Eisentraut wrote:
> Speaking of which, would it make sense to put this tool (whatever the name)
> into the pg_basebackup directory?  It's sort of related, and it also shares
> some code.

I've read the patch, and the additions to streamutil.h and
streamutil.c make it kind of natural to have it sit in pg_basebackup/.
There's pg_recvlogical already there.  I am wondering about two
things, though:
- Should the subdirectory pg_basebackup be renamed into something more
generic at this point?  All these things are frontend tools that deal
in some way with the replication protocol to do their work.  Say
a replication_tools?
- And if it would be better to refactor some of the code generic to
all these streaming tools to fe_utils.  What makes streamutil.h a bit
less pluggable are all its extern variables to control the connection,
but perhaps that can be an advantage, as well, in some cases.
--
Michael

---

From: "Euler Taveira" <[email-masked]>
Date: Wed, 08 Nov 2023 09:50:47 -0300

> > into the pg_basebackup directory?  It's sort of related, and it also shares

I used the CreateReplicationSlot() from streamutil.h but decided to use the
CREATE_REPLICATION_SLOT command directly because it needs the LSN as output. As
you noticed at that time I wouldn't like a dependency in the pg_basebackup
header files; if we move this binary to base backup directory, it seems natural
to refactor the referred function and use it.

> in some way with the replication protocol to do their work.  Say

It is a good fit for this tool since it is another replication tool. I also
agree with the directory renaming; it seems confusing that the directory has
the same name as one binary but also contains other related binaries in it.

> but perhaps that can be an advantage, as well, in some cases.

I like it. There are common functions such as GetConnection(),
CreateReplicationSlot(), DropReplicationSlot() and RunIdentifySystem() that is
used by all of these replication tools. We can move the extern variables into
parameters to have a pluggable streamutil.h.

--
Euler Taveira
EDB   https://www.enterprisedb.com/

---

From: Michael Paquier <[email-masked]>
Date: Thu, 9 Nov 2023 08:13:42 +0900

> to refactor the referred function and use it.

Right.  That should be OK to store that in an optional XLogRecPtr
pointer, aka by letting the option to pass NULL as argument of the
function if the caller needs nothing. 

> the same name as one binary but also contains other related binaries in it.

Or cluster_tools?  Or stream_tools?  replication_tools may be OK, but
I have a bad sense in naming new things around here.  So if anybody
has a better idea, feel free..

> parameters to have a pluggable streamutil.h.

And perhaps RetrieveWalSegSize() as well as GetSlotInformation().
These kick replication commands.
--
Michael

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 9 Nov 2023 15:41:53 +0100

> in some way with the replication protocol to do their work.  Say

Seems like unnecessary churn.  Nobody has complained about any of the 
other tools in there.

> but perhaps that can be an advantage, as well, in some cases.

Does anyone outside of pg_basebackup + existing friends + new friend 
need that?  Seems like extra complications.

---

From: Michael Paquier <[email-masked]>
Date: Fri, 10 Nov 2023 08:12:07 +0900

On Thu, Nov 09, 2023 at 03:41:53PM +0100, Peter Eisentraut wrote:
> Seems like unnecessary churn.  Nobody has complained about any of the other
> tools in there.

Not sure.  We rename things across releases in the tree from time to
time, and here that's straight-forward.

> Does anyone outside of pg_basebackup + existing friends + new friend need
> that?  Seems like extra complications.

Actually, yes, I've used these utility routines in some past work, and
having the wrapper routines able to run the replication commands in
fe_utils would have been nicer than having to link to a source tree.
--
Michael