MIME-Version: 1.0
References: <Z_1Cq7JvabsFYjQo@ark>
 <CAKAnmmJxGHcQmzXqEQ-nppZjXyH7i5W_7Cz8JNMb2+6c5+D+og@mail.gmail.com>
 <CAM527d9+eGvdih1GxoPDJ2WYs6eitf39oNTxGxnCK1PgF=A8wg@mail.gmail.com>
 <1798838.1744759182@sss.pgh.pa.us>
 <20FB597F-641F-48F8-8428-D8DDBA802D58@yandex-team.ru>
 <CAExHW5udz081Unx5qcLZ89DnR4cp6OhW0TCoco3+AEkGw480uQ@mail.gmail.com>
 <212D5973-FDD0-4CF5-BCD0-2760EC319DF3@yandex-team.ru>
 <CAExHW5tq8ShEfboFa52wDMWmyaW+6kX_6x8MZ-0TP65tOhJ1wQ@mail.gmail.com>
 <CAKAnmm+QJnL_APyKhM3hjkOv9DXjWaqJxB2nFDuNH9JqVdTfXA@mail.gmail.com>
 <0AA66026-3707-4252-B97A-86CC65F1F22E@upgrade.com>
 <610bfd9db7c29f03ba0d44d6fb9feb7c724f6f70.camel@gunduz.org>
In-Reply-To: <610bfd9db7c29f03ba0d44d6fb9feb7c724f6f70.camel@gunduz.org>
From: Jim Nasby <jnasby@upgrade.com>
Date: Tue, 29 Apr 2025 15:16:07 -0500
Message-ID: 
 <CAMFBP2oGM9N8Kty-3DzRmmgV0=ROB7ss7Rudw8Wida92t9O0+w@mail.gmail.com>
Subject: Re: Built-in Raft replication
To: =?UTF-8?B?RGV2cmltIEfDvG5kw7x6?= <devrim@gunduz.org>
Cc: Greg Sabino Mullane <htamfids@gmail.com>,
 Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
	Andrey Borodin <x4mmm@yandex-team.ru>, Tom Lane <tgl@sss.pgh.pa.us>,
	Konstantin Osipov <kostja.osipov@gmail.com>,
 Nikolay Samokhvalov <nik@postgres.ai>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="0000000000004cf2960633f07958"
Archived-At: 
 <https://www.postgresql.org/message-id/CAMFBP2oGM9N8Kty-3DzRmmgV0%3DROB7ss7Rudw8Wida92t9O0%2Bw%40mail.gmail.com>
Precedence: bulk

--0000000000004cf2960633f07958
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I've always assumed there'd have to be at least one global stream, if for
no other purpose than to be the source of truth about transaction commit
ordering (though, I was thinking of supporting multiple streams for one
database). Presumably the same could be used for shared objects. Or perhaps
shared objects just get their own stream. Either way, having a master
commit record that points at LSNs of various other streams is what I'd been
thinking.

On Wed, Apr 23, 2025 at 12:01=E2=80=AFPM Devrim G=C3=BCnd=C3=BCz <devrim@gu=
nduz.org> wrote:

> Hi,
>
> On Wed, 2025-04-23 at 11:48 -0500, Jim Nasby wrote:
> > unless we added multiple WAL streams. That would allow for splitting
> > WAL traffic across multiple devices as well as providing better
> > support for configurations that don=E2=80=99t replicate the entire clus=
ter.
> > The current situation where delayed replication of a single table
> > mandates retention of all the WAL for the entire cluster is less than
> > ideal.
>
> I think the problem is handling the stream of global objects. Having
> separate stream for each database would be awesome as long as it can
> deal with the "global stream".
>
> Regards,
> --
> Devrim G=C3=BCnd=C3=BCz
> Open Source Solution Architect, PostgreSQL Major Contributor
> BlueSky: @devrim.gunduz.org , @gunduz.org
>

--0000000000004cf2960633f07958
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I&#39;ve always assumed there&#39;d have to be at least on=
e global stream, if for no other purpose than to be the source of truth abo=
ut transaction commit ordering (though, I was thinking of supporting multip=
le streams for one database). Presumably the same could be used for shared =
objects. Or perhaps shared objects just get their own stream. Either way, h=
aving a master commit record that points at LSNs of various other streams i=
s what I&#39;d been thinking.</div><br><div class=3D"gmail_quote gmail_quot=
e_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Apr 23, 2025 at =
12:01=E2=80=AFPM Devrim G=C3=BCnd=C3=BCz &lt;<a href=3D"mailto:devrim@gundu=
z.org">devrim@gunduz.org</a>&gt; wrote:<br></div><blockquote class=3D"gmail=
_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204=
,204);padding-left:1ex">Hi,<br>
<br>
On Wed, 2025-04-23 at 11:48 -0500, Jim Nasby wrote:<br>
&gt; unless we added multiple WAL streams. That would allow for splitting<b=
r>
&gt; WAL traffic across multiple devices as well as providing better<br>
&gt; support for configurations that don=E2=80=99t replicate the entire clu=
ster.<br>
&gt; The current situation where delayed replication of a single table<br>
&gt; mandates retention of all the WAL for the entire cluster is less than<=
br>
&gt; ideal.<br>
<br>
I think the problem is handling the stream of global objects. Having<br>
separate stream for each database would be awesome as long as it can<br>
deal with the &quot;global stream&quot;.<br>
<br>
Regards,<br>
-- <br>
Devrim G=C3=BCnd=C3=BCz<br>
Open Source Solution Architect, PostgreSQL Major Contributor<br>
BlueSky: @<a href=3D"http://devrim.gunduz.org" rel=3D"noreferrer" target=3D=
"_blank">devrim.gunduz.org</a> , @<a href=3D"http://gunduz.org" rel=3D"nore=
ferrer" target=3D"_blank">gunduz.org</a><br>
</blockquote></div>

--0000000000004cf2960633f07958--