MIME-Version: 1.0
References: 
 <CALj2ACVfF2Uj9NoFy-5m98HNtjHpuD17EDE9twVeJng-jTAe7A@mail.gmail.com>
 <CAPsk3_CzLbMe-D07H5Vo6yWFvyXHh5is7AoPUCFcztrUmf1haw@mail.gmail.com>
 <CALj2ACVzL4uU=hxFpSfkqP4HeFCPucbBTEg6HNf_MPTYm52pHg@mail.gmail.com>
 <CAMm1aWYa1fGKcuG69xGJPNXLQ_9zHrPqhr-ZGdj4so6Exq66MQ@mail.gmail.com>
 <CALj2ACXa-2eEHHaNRwjcF1k9rtH=EJrWvbGJkucdSOD3zP-OUw@mail.gmail.com>
 <CAPsk3_A7079UtVqm2WXXiwadGJ7DucpenmLwnXZgDgXee703Rw@mail.gmail.com>
In-Reply-To: 
 <CAPsk3_A7079UtVqm2WXXiwadGJ7DucpenmLwnXZgDgXee703Rw@mail.gmail.com>
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sat, 13 Sep 2025 22:04:58 -0700
Message-ID: 
 <CALj2ACX+LKR7=3TkP83_9cdcXZd+9zhXWokjXyh5tTSi25+ogw@mail.gmail.com>
Subject: Re: Use WALReadFromBuffers in more places
To: Jingtang Zhang <mrdrivingduck@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org,
	Nitin Jadhav <nitinjadhavpostgres@gmail.com>
Content-Type: multipart/mixed; boundary="000000000000c318ae063ebbd4d2"
Archived-At: 
 <https://www.postgresql.org/message-id/CALj2ACX%2BLKR7%3D3TkP83_9cdcXZd%2B9zhXWokjXyh5tTSi25%2Bogw%40mail.gmail.com>
Precedence: bulk

--000000000000c318ae063ebbd4d2
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi,

On Tue, Oct 15, 2024 at 1:22=E2=80=AFAM Jingtang Zhang <mrdrivingduck@gmail=
.com> wrote:
>
> I've been back to this patch for a while recently. I witness that if a WA=
L
> writer works fast, the already flushed WAL buffers will be zeroed out and
> re-initialized for future use by AdvanceXLInsertBuffer in
> XLogBackgroundFlush, so that WALReadFromBuffers will miss even though the
> space of WAL buffer is enough. It is much more unfriendly for logical
> walsenders than physical walsenders, because logical ones consume WAL
> slower than physical ones due to the extra decoding phase. Seems that the=
 aim
> of AdvanceXLInsertBuffer in WAL writer contradicts with our reading from
> WAL buffer. Any thoughts?

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files. The no-locks design ensures it
never gets in the way of backends generating WAL, which is critical
for overall system performance.

I rebased and attached the v3 patch. I discarded the test extension
patch that demonstrated WALReadFromBuffers' behavior (i.e., waiting
for WAL to be fully copied to WAL buffers with
WaitXLogInsertionsToFinish), as I believe the comment at the top of
WALReadFromBuffers is sufficient documentation. I can reintroduce the
test extension if there's interest.

Thoughts?

--=20
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

--000000000000c318ae063ebbd4d2
Content-Type: application/octet-stream;
	name="v3-0001-Use-WALReadFromBuffers-in-more-places.patch"
Content-Disposition: attachment;
	filename="v3-0001-Use-WALReadFromBuffers-in-more-places.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mfj8bi4u0>
X-Attachment-Id: f_mfj8bi4u0

RnJvbSA2MGFhMmUxOGRmMDYwZmNjMzQxODQ3NTg1YWE4Y2Q0ZGNkZGVhNWVkIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBCaGFyYXRoIFJ1cGlyZWRkeSA8cnVwaXJlZGRAYW1hem9uLmNv
bT4KRGF0ZTogU3VuLCAxNCBTZXAgMjAyNSAwNDo1NDoxMiArMDAwMApTdWJqZWN0OiBbUEFUQ0gg
djNdIFVzZSBXQUxSZWFkRnJvbUJ1ZmZlcnMgaW4gbW9yZSBwbGFjZXMKCkNvbW1pdCA5MWYyY2Fl
IGludHJvZHVjZWQgV0FMUmVhZEZyb21CdWZmZXJzIGJ1dCB1c2VkIGl0IG9ubHkgZm9yCnBoeXNp
Y2FsIHJlcGxpY2F0aW9uIHdhbHNlbmRlcnMuIFRoZXJlIGFyZSBzZXZlcmFsIG90aGVyIGNhbGxl
cnMKdGhhdCB1c2UgdGhlIHJlYWRfbG9jYWxfeGxvZ19wYWdlIHBhZ2VfcmVhZCBjYWxsYmFjaywg
YW5kIGxvZ2ljYWwKcmVwbGljYXRpb24gd2Fsc2VuZGVycyBjYW4gYWxzbyBiZW5lZml0IGZyb20g
cmVhZGluZyBXQUwgZnJvbSBXQUwKYnVmZmVycyB1c2luZyB0aGUgbmV3IGZ1bmN0aW9uLiBUaGlz
IGNvbW1pdCBleHRlbmRzIHRoZSB1c2Ugb2YKV0FMUmVhZEZyb21CdWZmZXJzIHRvIHRoZXNlIGNh
bGxlcnMuCgpBdXRob3I6IEJoYXJhdGggUnVwaXJlZGR5ClJldmlld2VkLWJ5OiBKaW5ndGFuZyBa
aGFuZywgTml0aW4gSmFkaGF2CkRpc2N1c3Npb246IGh0dHBzOi8vd3d3LnBvc3RncmVzcWwub3Jn
L21lc3NhZ2UtaWQvQ0FMajJBQ1ZmRjJVajlOb0Z5LTVtOThITnRqSHB1RDE3RURFOXR3VmVKbmct
alRBZTdBJTQwbWFpbC5nbWFpbC5jb20KLS0tCiBzcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94
bG9ndXRpbHMuYyB8IDIzICsrKysrKystCiBzcmMvYmFja2VuZC9yZXBsaWNhdGlvbi93YWxzZW5k
ZXIuYyAgICB8IDc3ICsrKysrKysrKysrKysrKysrLS0tLS0tLS0tCiAyIGZpbGVzIGNoYW5nZWQs
IDcwIGluc2VydGlvbnMoKyksIDMwIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3NyYy9iYWNr
ZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2d1dGlscy5jIGIvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5z
YW0veGxvZ3V0aWxzLmMKaW5kZXggMzgxNzZkOTY4OGUuLjNlNWI0Zjc1ZmViIDEwMDY0NAotLS0g
YS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9ndXRpbHMuYworKysgYi9zcmMvYmFja2Vu
ZC9hY2Nlc3MvdHJhbnNhbS94bG9ndXRpbHMuYwpAQCAtODc2LDYgKzg3Niw3IEBAIHJlYWRfbG9j
YWxfeGxvZ19wYWdlX2d1dHMoWExvZ1JlYWRlclN0YXRlICpzdGF0ZSwgWExvZ1JlY1B0ciB0YXJn
ZXRQYWdlUHRyLAogCWludAkJCWNvdW50OwogCVdBTFJlYWRFcnJvciBlcnJpbmZvOwogCVRpbWVM
aW5lSUQJY3VyclRMSTsKKwlTaXplCQlieXRlc1JlYWQ7CiAKIAlsb2MgPSB0YXJnZXRQYWdlUHRy
ICsgcmVxTGVuOwogCkBAIC05OTUsOSArOTk2LDI1IEBAIHJlYWRfbG9jYWxfeGxvZ19wYWdlX2d1
dHMoWExvZ1JlYWRlclN0YXRlICpzdGF0ZSwgWExvZ1JlY1B0ciB0YXJnZXRQYWdlUHRyLAogCQlj
b3VudCA9IHJlYWRfdXB0byAtIHRhcmdldFBhZ2VQdHI7CiAJfQogCi0JaWYgKCFXQUxSZWFkKHN0
YXRlLCBjdXJfcGFnZSwgdGFyZ2V0UGFnZVB0ciwgY291bnQsIHRsaSwKLQkJCQkgJmVycmluZm8p
KQotCQlXQUxSZWFkUmFpc2VFcnJvcigmZXJyaW5mbyk7CisJLyogRmlyc3QgYXR0ZW1wdCB0byBy
ZWFkIGZyb20gV0FMIGJ1ZmZlcnMgKi8KKwlieXRlc1JlYWQgPSBXQUxSZWFkRnJvbUJ1ZmZlcnMo
Y3VyX3BhZ2UsIHRhcmdldFBhZ2VQdHIsIGNvdW50LCBjdXJyVExJKTsKKworCS8qIElmIHdlIHN0
aWxsIGhhdmUgYnl0ZXMgdG8gcmVhZCwgZ2V0IHRoZW0gZnJvbSBXQUwgZmlsZSAqLworCWlmIChi
eXRlc1JlYWQgPCBjb3VudCkKKwl7CisJCWlmICghV0FMUmVhZChzdGF0ZSwKKwkJCQkJIGN1cl9w
YWdlICsgYnl0ZXNSZWFkLAorCQkJCQkgdGFyZ2V0UGFnZVB0ciArIGJ5dGVzUmVhZCwKKwkJCQkJ
IGNvdW50IC0gYnl0ZXNSZWFkLAorCQkJCQkgdGxpLAorCQkJCQkgJmVycmluZm8pKQorCQl7CisJ
CQlXQUxSZWFkUmFpc2VFcnJvcigmZXJyaW5mbyk7CisJCX0KKwkJYnl0ZXNSZWFkID0gY291bnQ7
CQkvKiBBbGwgcmVxdWVzdGVkIGJ5dGVzIHJlYWQgKi8KKwl9CisKKwlBc3NlcnQoYnl0ZXNSZWFk
ID09IGNvdW50KTsKIAogCS8qIG51bWJlciBvZiB2YWxpZCBieXRlcyBpbiB0aGUgYnVmZmVyICov
CiAJcmV0dXJuIGNvdW50OwpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vd2Fs
c2VuZGVyLmMgYi9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi93YWxzZW5kZXIuYwppbmRleCA1OTgy
MmYyMmI4ZC4uOTM3OTM2YzM1NTAgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9u
L3dhbHNlbmRlci5jCisrKyBiL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL3dhbHNlbmRlci5jCkBA
IC0xMDM2LDYgKzEwMzYsNyBAQCBsb2dpY2FsX3JlYWRfeGxvZ19wYWdlKFhMb2dSZWFkZXJTdGF0
ZSAqc3RhdGUsIFhMb2dSZWNQdHIgdGFyZ2V0UGFnZVB0ciwgaW50IHJlcQogCVdBTFJlYWRFcnJv
ciBlcnJpbmZvOwogCVhMb2dTZWdObwlzZWdubzsKIAlUaW1lTGluZUlECWN1cnJUTEk7CisJU2l6
ZQkJYnl0ZXNSZWFkOwogCiAJLyoKIAkgKiBNYWtlIHN1cmUgd2UgaGF2ZSBlbm91Z2ggV0FMIGF2
YWlsYWJsZSBiZWZvcmUgcmV0cmlldmluZyB0aGUgY3VycmVudApAQCAtMTA3MywxNiArMTA3NCwy
OSBAQCBsb2dpY2FsX3JlYWRfeGxvZ19wYWdlKFhMb2dSZWFkZXJTdGF0ZSAqc3RhdGUsIFhMb2dS
ZWNQdHIgdGFyZ2V0UGFnZVB0ciwgaW50IHJlcQogCWVsc2UKIAkJY291bnQgPSBmbHVzaHB0ciAt
IHRhcmdldFBhZ2VQdHI7CS8qIHBhcnQgb2YgdGhlIHBhZ2UgYXZhaWxhYmxlICovCiAKLQkvKiBu
b3cgYWN0dWFsbHkgcmVhZCB0aGUgZGF0YSwgd2Uga25vdyBpdCdzIHRoZXJlICovCi0JaWYgKCFX
QUxSZWFkKHN0YXRlLAotCQkJCSBjdXJfcGFnZSwKLQkJCQkgdGFyZ2V0UGFnZVB0ciwKLQkJCQkg
Y291bnQsCi0JCQkJIGN1cnJUTEksCQkvKiBQYXNzIHRoZSBjdXJyZW50IFRMSSBiZWNhdXNlIG9u
bHkKKwkvKiBGaXJzdCBhdHRlbXB0IHRvIHJlYWQgZnJvbSBXQUwgYnVmZmVycyAqLworCWJ5dGVz
UmVhZCA9IFdBTFJlYWRGcm9tQnVmZmVycyhjdXJfcGFnZSwgdGFyZ2V0UGFnZVB0ciwgY291bnQs
IGN1cnJUTEkpOworCisJdGFyZ2V0UGFnZVB0ciArPSBieXRlc1JlYWQ7CisKKwkvKiBJZiB3ZSBz
dGlsbCBoYXZlIGJ5dGVzIHRvIHJlYWQsIGdldCB0aGVtIGZyb20gV0FMIGZpbGUgKi8KKwlpZiAo
Ynl0ZXNSZWFkIDwgY291bnQpCisJeworCQlpZiAoIVdBTFJlYWQoc3RhdGUsCisJCQkJCSBjdXJf
cGFnZSArIGJ5dGVzUmVhZCwKKwkJCQkJIHRhcmdldFBhZ2VQdHIsCisJCQkJCSBjb3VudCAtIGJ5
dGVzUmVhZCwKKwkJCQkJIGN1cnJUTEksCS8qIFBhc3MgdGhlIGN1cnJlbnQgVExJIGJlY2F1c2Ug
b25seQogCQkJCQkJCQkgKiBXYWxTbmRTZWdtZW50T3BlbiBjb250cm9scyB3aGV0aGVyIG5ldyBU
TEkKIAkJCQkJCQkJICogaXMgbmVlZGVkLiAqLwotCQkJCSAmZXJyaW5mbykpCi0JCVdBTFJlYWRS
YWlzZUVycm9yKCZlcnJpbmZvKTsKKwkJCQkJICZlcnJpbmZvKSkKKwkJeworCQkJV0FMUmVhZFJh
aXNlRXJyb3IoJmVycmluZm8pOworCQl9CisJCWJ5dGVzUmVhZCA9IGNvdW50OwkJLyogQWxsIHJl
cXVlc3RlZCBieXRlcyByZWFkICovCisJfQorCisJQXNzZXJ0KGJ5dGVzUmVhZCA9PSBjb3VudCk7
CiAKIAkvKgogCSAqIEFmdGVyIHJlYWRpbmcgaW50byB0aGUgYnVmZmVyLCBjaGVjayB0aGF0IHdo
YXQgd2UgcmVhZCB3YXMgdmFsaWQuIFdlIGRvCkBAIC0zMTc2LDcgKzMxOTAsNyBAQCBYTG9nU2Vu
ZFBoeXNpY2FsKHZvaWQpCiAJU2l6ZQkJbmJ5dGVzOwogCVhMb2dTZWdObwlzZWdubzsKIAlXQUxS
ZWFkRXJyb3IgZXJyaW5mbzsKLQlTaXplCQlyYnl0ZXM7CisJU2l6ZQkJYnl0ZXNSZWFkOwogCiAJ
LyogSWYgcmVxdWVzdGVkIHN3aXRjaCB0aGUgV0FMIHNlbmRlciB0byB0aGUgc3RvcHBpbmcgc3Rh
dGUuICovCiAJaWYgKGdvdF9TVE9QUElORykKQEAgLTMzOTIsMjQgKzM0MDYsMzMgQEAgWExvZ1Nl
bmRQaHlzaWNhbCh2b2lkKQogCWVubGFyZ2VTdHJpbmdJbmZvKCZvdXRwdXRfbWVzc2FnZSwgbmJ5
dGVzKTsKIAogcmV0cnk6Ci0JLyogYXR0ZW1wdCB0byByZWFkIFdBTCBmcm9tIFdBTCBidWZmZXJz
IGZpcnN0ICovCi0JcmJ5dGVzID0gV0FMUmVhZEZyb21CdWZmZXJzKCZvdXRwdXRfbWVzc2FnZS5k
YXRhW291dHB1dF9tZXNzYWdlLmxlbl0sCi0JCQkJCQkJCXN0YXJ0cHRyLCBuYnl0ZXMsIHhsb2dy
ZWFkZXItPnNlZy53c190bGkpOwotCW91dHB1dF9tZXNzYWdlLmxlbiArPSByYnl0ZXM7Ci0Jc3Rh
cnRwdHIgKz0gcmJ5dGVzOwotCW5ieXRlcyAtPSByYnl0ZXM7Ci0KLQkvKiBub3cgcmVhZCB0aGUg
cmVtYWluaW5nIFdBTCBmcm9tIFdBTCBmaWxlICovCi0JaWYgKG5ieXRlcyA+IDAgJiYKLQkJIVdB
TFJlYWQoeGxvZ3JlYWRlciwKLQkJCQkgJm91dHB1dF9tZXNzYWdlLmRhdGFbb3V0cHV0X21lc3Nh
Z2UubGVuXSwKLQkJCQkgc3RhcnRwdHIsCi0JCQkJIG5ieXRlcywKLQkJCQkgeGxvZ3JlYWRlci0+
c2VnLndzX3RsaSwJLyogUGFzcyB0aGUgY3VycmVudCBUTEkgYmVjYXVzZQotCQkJCQkJCQkJCQkg
KiBvbmx5IFdhbFNuZFNlZ21lbnRPcGVuIGNvbnRyb2xzCi0JCQkJCQkJCQkJCSAqIHdoZXRoZXIg
bmV3IFRMSSBpcyBuZWVkZWQuICovCi0JCQkJICZlcnJpbmZvKSkKLQkJV0FMUmVhZFJhaXNlRXJy
b3IoJmVycmluZm8pOworCS8qIEZpcnN0IGF0dGVtcHQgdG8gcmVhZCBmcm9tIFdBTCBidWZmZXJz
ICovCisJYnl0ZXNSZWFkID0gV0FMUmVhZEZyb21CdWZmZXJzKCZvdXRwdXRfbWVzc2FnZS5kYXRh
W291dHB1dF9tZXNzYWdlLmxlbl0sCisJCQkJCQkJCSAgIHN0YXJ0cHRyLAorCQkJCQkJCQkgICBu
Ynl0ZXMsCisJCQkJCQkJCSAgIHhsb2dyZWFkZXItPnNlZy53c190bGkpOworCisJc3RhcnRwdHIg
Kz0gYnl0ZXNSZWFkOworCisJLyogSWYgd2Ugc3RpbGwgaGF2ZSBieXRlcyB0byByZWFkLCBnZXQg
dGhlbSBmcm9tIFdBTCBmaWxlICovCisJaWYgKGJ5dGVzUmVhZCA8IG5ieXRlcykKKwl7CisJCWlm
ICghV0FMUmVhZCh4bG9ncmVhZGVyLAorCQkJCQkgJm91dHB1dF9tZXNzYWdlLmRhdGFbb3V0cHV0
X21lc3NhZ2UubGVuICsgYnl0ZXNSZWFkXSwKKwkJCQkJIHN0YXJ0cHRyLAorCQkJCQkgbmJ5dGVz
IC0gYnl0ZXNSZWFkLAorCQkJCQkgeGxvZ3JlYWRlci0+c2VnLndzX3RsaSwJLyogUGFzcyB0aGUg
Y3VycmVudCBUTEkKKwkJCQkJCQkJCQkJCSAqIGJlY2F1c2Ugb25seQorCQkJCQkJCQkJCQkJICog
V2FsU25kU2VnbWVudE9wZW4gY29udHJvbHMKKwkJCQkJCQkJCQkJCSAqIHdoZXRoZXIgbmV3IFRM
SSBpcyBuZWVkZWQuICovCisJCQkJCSAmZXJyaW5mbykpCisJCXsKKwkJCVdBTFJlYWRSYWlzZUVy
cm9yKCZlcnJpbmZvKTsKKwkJfQorCQlieXRlc1JlYWQgPSBuYnl0ZXM7CQkvKiBBbGwgcmVxdWVz
dGVkIGJ5dGVzIHJlYWQgKi8KKwl9CisKKwlBc3NlcnQoYnl0ZXNSZWFkID09IG5ieXRlcyk7CiAK
IAkvKiBTZWUgbG9naWNhbF9yZWFkX3hsb2dfcGFnZSgpLiAqLwogCVhMQnl0ZVRvU2VnKHN0YXJ0
cHRyLCBzZWdubywgeGxvZ3JlYWRlci0+c2VnY3h0LndzX3NlZ3NpemUpOwotLSAKMi40Ny4zCgo=
--000000000000c318ae063ebbd4d2--