MIME-Version: 1.0
References: 
 <CALj2ACVfF2Uj9NoFy-5m98HNtjHpuD17EDE9twVeJng-jTAe7A@mail.gmail.com>
 <CAPsk3_CzLbMe-D07H5Vo6yWFvyXHh5is7AoPUCFcztrUmf1haw@mail.gmail.com>
 <CALj2ACVzL4uU=hxFpSfkqP4HeFCPucbBTEg6HNf_MPTYm52pHg@mail.gmail.com>
 <CAMm1aWYa1fGKcuG69xGJPNXLQ_9zHrPqhr-ZGdj4so6Exq66MQ@mail.gmail.com>
 <CALj2ACXa-2eEHHaNRwjcF1k9rtH=EJrWvbGJkucdSOD3zP-OUw@mail.gmail.com>
 <CAPsk3_A7079UtVqm2WXXiwadGJ7DucpenmLwnXZgDgXee703Rw@mail.gmail.com>
 <CALj2ACX+LKR7=3TkP83_9cdcXZd+9zhXWokjXyh5tTSi25+ogw@mail.gmail.com>
 <fa2e932eeff472250e2dbacb49d8c43ad282fea9.camel@j-davis.com>
In-Reply-To: <fa2e932eeff472250e2dbacb49d8c43ad282fea9.camel@j-davis.com>
From: Rahila Syed <rahilasyed90@gmail.com>
Date: Wed, 24 Sep 2025 10:31:29 +0530
Message-ID: 
 <CAH2L28uvqq_WxZqFJYXUxqZzXmfDC6gP9uYLQXmYMtFq+7Jm_A@mail.gmail.com>
Subject: Re: Use WALReadFromBuffers in more places
To: Jeff Davis <pgsql@j-davis.com>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	pgsql-hackers@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000c99412063f84f20d"
Archived-At: 
 <https://www.postgresql.org/message-id/CAH2L28uvqq_WxZqFJYXUxqZzXmfDC6gP9uYLQXmYMtFq%2B7Jm_A%40mail.gmail.com>
Precedence: bulk

--000000000000c99412063f84f20d
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi,

On Mon, Sep 22, 2025 at 8:56=E2=80=AFPM Jeff Davis <pgsql@j-davis.com> wrot=
e:

> On Sat, 2025-09-13 at 22:04 -0700, Bharath Rupireddy wrote:
> > Thanks for looking at this. Yes, the WAL writers can zero out flushed
> > buffers before WALReadFromBuffers gets to them. However,
> > WALReadFromBuffers was intentionally designed as an opportunistic
> > optimization - it's a "try this first, quickly" approach before
> > falling back to reading from WAL files.
>
> IIRC, one motivation (perhaps the primary motivation?) was to make it
> possible to read buffers before they are flushed. It was always
> possible to read already-flushed buffers.
>
The benefit of reading unflushed buffers is that we can replicate the
> WAL sooner (though it can't be replayed until the primary flushes it).
> Is that right?
>

 I'm not certain about the primary motivation, but as it stands,
WALReadFromBuffers only reads WAL records present in buffers up to the
flush pointer. This is because XLogSendPhysical currently sends records
only up to the flush pointer, not beyond.

I am currently testing a patch developed by Melih Mutlu that implements the
functionality you described, sending unflushed buffers during physical
replication. After some tuning, the patch has shown a 5 percent improvement
in TPS for synchronous replication with remote_write. I am working on
further improving the patch before sharing it on the hackers mailing list.

Thank you,
Rahila Syed

--000000000000c99412063f84f20d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hi,</div><br><div class=3D"gmail_quote gm=
ail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Sep 22, =
2025 at 8:56=E2=80=AFPM Jeff Davis &lt;<a href=3D"mailto:pgsql@j-davis.com"=
>pgsql@j-davis.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote=
" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);=
padding-left:1ex">On Sat, 2025-09-13 at 22:04 -0700, Bharath Rupireddy wrot=
e:<br>
&gt; Thanks for looking at this. Yes, the WAL writers can zero out flushed<=
br>
&gt; buffers before WALReadFromBuffers gets to them. However,<br>
&gt; WALReadFromBuffers was intentionally designed as an opportunistic<br>
&gt; optimization - it&#39;s a &quot;try this first, quickly&quot; approach=
 before<br>
&gt; falling back to reading from WAL files.<br>
<br>
IIRC, one motivation (perhaps the primary motivation?) was to make it<br>
possible to read buffers before they are flushed. It was always<br>
possible to read already-flushed buffers.=C2=A0<br></blockquote><blockquote=
 class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px so=
lid rgb(204,204,204);padding-left:1ex">
The benefit of reading unflushed buffers is that we can replicate the<br>
WAL sooner (though it can&#39;t be replayed until the primary flushes it).<=
br>
Is that right?<br></blockquote><div><br>=C2=A0I&#39;m not certain about the=
 primary motivation, but as it stands, WALReadFromBuffers only reads WAL re=
cords present in buffers up to the flush pointer. This is because XLogSendP=
hysical currently sends records only up to the flush pointer, not beyond.=
=C2=A0=C2=A0<br><br>I am currently testing a patch developed by Melih Mutlu=
 that implements the functionality you described, sending unflushed buffers=
 during physical replication. After some tuning, the patch has shown a 5 pe=
rcent improvement in TPS for synchronous replication with remote_write. I a=
m working on further improving the patch before sharing it on the hackers m=
ailing list.<br><br>Thank you,</div><div>Rahila Syed</div></div></div>

--000000000000c99412063f84f20d--