Subject: Re: [HACKERS] Logical replication and multimaster
From: =?UTF-8?Q?Jon_Erdman?= <jon@thewickedtribe.net>
To: =?UTF-8?Q?Jim_Nasby?= <Jim.Nasby@BlueTreble.com>, 
	=?UTF-8?Q?David_Fetter?= <david@fetter.org>, 
	=?UTF-8?Q?Craig_Ringer?= <craig@2ndquadrant.com>
Cc: =?UTF-8?Q?Robert_Haas?= <robertmhaas@gmail.com>, 
	=?UTF-8?Q?Andres_Freund?= <andres@anarazel.de>, 
	=?UTF-8?Q?Simon_Riggs?= <simon@2ndquadrant.com>, 
	=?UTF-8?Q?Konstantin_Knizhnik?= <k.knizhnik@postgrespro.ru>, 
	=?UTF-8?Q?pgsql-hackers?= <pgsql-hackers@postgresql.org>, 
	=?UTF-8?Q?Tom_Lane?= <tgl@sss.pgh.pa.us>, 
	=?UTF-8?Q?Jon_Erdman?= <jon@thewickedtribe.net>
Date: Thu, 5 Oct 2023 05:31:00 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <20d5abfe-0fdd-47dc-8d67-7288c650a147@thewickedtribe.net>
References: <565C7756.9070604@postgrespro.ru> 
 <CANP8+jLmPXKxDsqXwf6RNSp9Q-6uc7UcZo=KNfCzH0PeehhE=w@mail.gmail.com> 
 <CAMsr+YEfTEp35bQ5ia__nDwc2MwzYUQcuQ__vT_b=UNOHyooMA@mail.gmail.com> 
 <CA+TgmoaqJJsPNb7JDAHTtGZbH7CSenuf=0trQ_j1+_scLE-c0w@mail.gmail.com> 
 <CAMsr+YFf4hA=hLTe2hvPZ95-Fxf3bT80-_8CWwrDpDzva3LTOw@mail.gmail.com> 
 <20151211101608.GN14789@awork2.anarazel.de> 
 <CA+TgmobiQUpXpiJ0z+=PMjL4at5TteZ27A8JPMY+QQb_9NeyJA@mail.gmail.com> 
 <CAMsr+YF7vP72LzMmNhe0ppZg-KnEt8d_RXzyjZaZMNykmQEiVQ@mail.gmail.com> 
 <20151213133737.GA28490@fetter.org> <5670DF24.7090101@BlueTreble.com> 
 <20d5abfe-0fdd-47dc-8d67-7288c650a147@thewickedtribe.net> 
 <bf676ae1-3b9a-a40c-412f-3cc84516ec31@thewickedtribe.net>
Thread-Index: AQHY2FQMdriFB/pPR6mKeY1wtCSc2Ee9saCSR75Es6E=
Thread-Topic: [HACKERS] Logical replication and multimaster
Message-ID: <0101018afe515ec9-a3b31eeb-8040-4c9d-a8e3-02cf0c7ed197-000000@us-west-2.amazonses.com>
Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES
Archived-At: <https://www.postgresql.org/message-id/0101018afe515ec9-a3b31eeb-8040-4c9d-a8e3-02cf0c7ed197-000000%40us-west-2.amazonses.com>
Precedence: bulk

=0D=0AOops,=0D=0A=0D=0ALooking at my old message from 2015 on this subjec=
t, it was Magnus who=20=0D=0Apish-poshed it, and Page who liked it. Don't=
 want to throw shaded at the=20=0D=0Awrong person ;)=0D=0A--=0D=0AJon Erd=
man (aka StuckMojo)=0D=0A     PostgreSQL Zealot=0D=0A=0D=0AOn 10/5/23 12:=
14 AM, Jon Erdman wrote:=0D=0A>=20=0D=0A> Well,=0D=0A>=20=0D=0A> Given my=
 earlier message about Apple wanting to pay me for writing=20=0D=0A> patc=
hes now, maybe I can revisit this idea.=0D=0A>=20=0D=0A> For background: =
I brought up the idea of an FDW that could read custom=20=0D=0A> dump fil=
es and expose them as tables so that you could grab just a=20=0D=0A> sing=
le record (or of course more) from a custom dump file without having=20=0D=
=0A> to load the whole thing up, if you were stuck reaching into a backup=
 to=20=0D=0A> get at accidentally deleted tables, rows, etc.. The stopper=
, which was=20=0D=0A> communicated to me by Tom at the following pgcon wa=
s that the code for=20=0D=0A> parsing custom dumps is duplicated in pg_du=
mp only, and would need to be=20=0D=0A> duplicated into the server for th=
e FDW, or broken out into a library.=0D=0A>=20=0D=0A> And for posterity, =
Dave Page said that was a stupid idea, while Magnus=20=0D=0A> said that i=
t sounded useful. And IIRC Bruce and Robert H said it was=20=0D=0A> doabl=
e, just a good deal of work on the refactor needed.=0D=0A>=20=0D=0A> This=
 convo went down at the Amsterdam conf where I spoke about using=20=0D=0A=
> writeable LVM snapshots to expose each developer a copy of prod to=20=0D=
=0A> noodle on, without having to actually make a full copy for each dev.=
=0D=0A>=20=0D=0A> Added trivia: I gave the talk with a can of Heineken in=
 my hand at the=20=0D=0A> podium, and my lightning talk had the work F(&k=
ing Cool in the title ;)=0D=0A>=20=0D=0A> That was also when I bought my =
plush Slony which was supposedly the very=20=0D=0A> last one. (turns out =
they made more)=0D=0A> --=20=0D=0A> Jon Erdman (aka StuckMojo)=0D=0A>  =C2=
=A0=C2=A0=C2=A0 PostgreSQL Zealot=0D=0A>=20=0D=0A> On 12/15/15 9:48 PM, J=
im Nasby wrote:=0D=0A>> On 12/13/15 7:37 AM, David Fetter wrote:=0D=0A>>>=
 As I understand it, pushing these into a library has been proposed but=0D=
=0A>>> not rejected.=C2=A0 That it hasn't happened yet is mostly about th=
e lack of=0D=0A>>> tuits (the round ones) to rewrite the functionality as=
 libraries and=0D=0A>>> refactor pg_dump/pg_restore to use only calls to =
same.=C2=A0 As usual, it's=0D=0A>>> less about writing the code and more =
about the enormous amount of=0D=0A>>> testing any such a refactor would e=
ntail.=0D=0A>>=0D=0A>> My understanding as well. IIRC Jon Erdman brought =
this question up a=20=0D=0A>> couple years ago and the response was "It'd=
 probably be accepted, it's=20=0D=0A>> just that no one has done the work=
=2E"=0D=0A>>=0D=0A>>> I believe that refactoring much of pg_dump's functi=
onality for the=0D=0A>>> current version of the server into SQL-accessibl=
e functions and making=0D=0A>>> pg_dump use only those functions is achie=
vable with available=0D=0A>>> resources.=0D=0A>>>=0D=0A>>> Such a refacto=
r need not be all-or-nothing.=C2=A0 For example, the=0D=0A>>> dependency =
resolution stuff is a first step that appears to be worth=0D=0A>>> doing =
by itself even if the effort then pauses, possibly for some=0D=0A>>> time=
=2E=0D=0A>>=0D=0A>> If someone wanted to spend time on this, I suspect it=
'd be worth=20=0D=0A>> looking at how bad some of the backward compatibil=
ity issues would be=20=0D=0A>> if done in the server. Maybe they wouldn't=
 be that bad. I suspect the=20=0D=0A>> audience for this code would be mu=
ch larger if it was in the server as=20=0D=0A>> opposed to a C library.=0D=
=0A