Content-Type: multipart/mixed; boundary="------------ekWnxcRaZg9zY3cB20IsJq2k"
Message-ID: <1f2ad9fd-53ff-74cc-a602-72d45423bcab@gmail.com>
Date: Wed, 21 Dec 2022 10:06:42 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.6.0
Subject: Re: Minimal logical decoding on standbys
To: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>,
 Ibrar Ahmed <ibrar.ahmad@gmail.com>, Amit Khandekar
 <amitdkhan.pg@gmail.com>, fabriziomello@gmail.com,
 tushar <tushar.ahuja@enterprisedb.com>,
 Rahila Syed <rahila.syed@2ndquadrant.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
References: <4b19419f-2e7c-a76b-c948-26d10af6693d@gmail.com>
 <20221207175824.apmc5clnj2fdjkaq@awork3.anarazel.de>
 <fe81c2e4-aeca-4e57-b453-f183531ce389@gmail.com>
 <09c26df0-c5f9-eae0-5a6a-10fb00b0522e@gmail.com>
 <CA+TgmoY0df9X+5ENg8P0BGj0odhM45sdQ7kB4JMo4NKaoFy-Vg@mail.gmail.com>
 <9f0a021e-40e1-0ae6-8086-a4457c99749e@gmail.com>
 <CA+Tgmoa1cZSo1tUTzxS+RyniMuNtkpxUeEhnD2+-mvJodVtdSQ@mail.gmail.com>
 <0ae83e69-bda0-0eb8-ef4a-d169292e3f66@gmail.com>
 <47f5faf0-6a6d-602a-5d5b-9449ca74c6ad@gmail.com>
 <CA+TgmoaVcu_mbxbH=EccvKG6u8+MdQf9zx98uAL9zsStFwrYsQ@mail.gmail.com>
 <20221214173552.sshklqvwezeo52h3@alap3.anarazel.de>
 <CA+TgmoZ8-DXtDMMDGggFkSYyO8nmNdBKWjV6Ts-zYGR9p5sK5g@mail.gmail.com>
 <CA+TgmobmTvEUDVOBzARa9eD9XLqsvDW+v29dV+WvDKZPJAwNTQ@mail.gmail.com>
 <CA+TgmoaOsoQuXEo6gMq0GP8SyvCaS=ptYqdYc0uzDMoeShf_nw@mail.gmail.com>
 <CA+TgmoYTTsxP8y6uknZvCBNCRq+1FJ4zGbX8Px1TGW459fGsaQ@mail.gmail.com>
Content-Language: en-US
From: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com>
In-Reply-To: 
 <CA+TgmoYTTsxP8y6uknZvCBNCRq+1FJ4zGbX8Px1TGW459fGsaQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/1f2ad9fd-53ff-74cc-a602-72d45423bcab%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------ekWnxcRaZg9zY3cB20IsJq2k
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

On 12/20/22 10:41 PM, Robert Haas wrote:
> On Tue, Dec 20, 2022 at 3:39 PM Robert Haas <robertmhaas@gmail.com> wrote:
>> I think this might be the only WAL record type where there's a
>> problem, but I haven't fully confirmed that yet.
> 
> It's not. GIST has the same issue. The same test case demonstrates the
> problem there, if you substitute this test script for
> kpt_hash_setup.sql and possibly also run it for somewhat longer. One
> might think that this wouldn't be a problem, because the comments for
> gistxlogDelete say this:
> 
>       /*
>        * In payload of blk 0 : todelete OffsetNumbers
>        */
> 
> But it's not in the payload of blk 0. It follows the main payload.

Oh right, nice catch!

Indeed, we can see in gistRedoDeleteRecord():

"
todelete = (OffsetNumber *) ((char *) xldata + SizeOfGistxlogDelete);
"

> 
> This is the reverse of xl_heap_freeze_page, which claims that freeze
> plans and offset numbers follow, but they don't: they're in the data
> for block 0. 

oh right, we can see in heap_xlog_freeze_page():

"
		plans = (xl_heap_freeze_plan *) XLogRecGetBlockData(record, 0, NULL);
		offsets = (OffsetNumber *) ((char *) plans +
									(xlrec->nplans *
									 sizeof(xl_heap_freeze_plan)));
"


> xl_btree_delete is also wrong, claiming that the data
> follows when it's really attached to block 0. 


oh right, we can see in btree_xlog_delete():

"
		char	   *ptr = XLogRecGetBlockData(record, 0, NULL);

		page = (Page) BufferGetPage(buffer);

		if (xlrec->nupdated > 0)
		{
			OffsetNumber *updatedoffsets;
			xl_btree_update *updates;

			updatedoffsets = (OffsetNumber *)
				(ptr + xlrec->ndeleted * sizeof(OffsetNumber));
			updates = (xl_btree_update *) ((char *) updatedoffsets +
										   xlrec->nupdated *
										   sizeof(OffsetNumber));
"


> I guess whatever else we
> do here, we should fix the comments.
> 

Agree, please find attached a patch proposal doing so.


> Bottom line is that I think the two cases that have alignment issues
> as coded are xl_hash_vacuum_one_page and gistxlogDelete. Everything
> else is OK, as far as I can tell right now.
> 

Thanks a lot for the repro(s) and explanations! That's very useful/helpful.

Based on your discovery about the wrong comments above, I'm now tempted to fix those 2 alignment issues
by using a FLEXIBLE_ARRAY_MEMBER within those structs (as you proposed in [1]) (as that should also prevent
any possible wrong comments about where the array is located).

What do you think?

[1]: https://www.postgresql.org/message-id/CA%2BTgmoaVcu_mbxbH%3DEccvKG6u8%2BMdQf9zx98uAL9zsStFwrYsQ%40mail.gmail.com

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
--------------ekWnxcRaZg9zY3cB20IsJq2k
Content-Type: text/plain; charset=UTF-8;
 name="v1-0001-fix-some-comments.patch"
Content-Disposition: attachment; filename="v1-0001-fix-some-comments.patch"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRlL2FjY2Vzcy9naXN0eGxvZy5oIGIvc3JjL2luY2x1
ZGUvYWNjZXNzL2dpc3R4bG9nLmgKaW5kZXggMzNmMWM3ZTMxYi4uOTUwNjhmZWI4NyAxMDA2
NDQKLS0tIGEvc3JjL2luY2x1ZGUvYWNjZXNzL2dpc3R4bG9nLmgKKysrIGIvc3JjL2luY2x1
ZGUvYWNjZXNzL2dpc3R4bG9nLmgKQEAgLTUyLDkgKzUyLDcgQEAgdHlwZWRlZiBzdHJ1Y3Qg
Z2lzdHhsb2dEZWxldGUKIAlUcmFuc2FjdGlvbklkIHNuYXBzaG90Q29uZmxpY3RIb3Jpem9u
OwogCXVpbnQxNgkJbnRvZGVsZXRlOwkJLyogbnVtYmVyIG9mIGRlbGV0ZWQgb2Zmc2V0cyAq
LwogCi0JLyoKLQkgKiBJbiBwYXlsb2FkIG9mIGJsayAwIDogdG9kZWxldGUgT2Zmc2V0TnVt
YmVycwotCSAqLworCS8qIFRPREVMRVRFIE9GRlNFVCBOVU1CRVIgQVJSQVkgRk9MTE9XUyAq
LwogfSBnaXN0eGxvZ0RlbGV0ZTsKIAogI2RlZmluZSBTaXplT2ZHaXN0eGxvZ0RlbGV0ZQko
b2Zmc2V0b2YoZ2lzdHhsb2dEZWxldGUsIG50b2RlbGV0ZSkgKyBzaXplb2YodWludDE2KSkK
ZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRlL2FjY2Vzcy9oZWFwYW1feGxvZy5oIGIvc3JjL2lu
Y2x1ZGUvYWNjZXNzL2hlYXBhbV94bG9nLmgKaW5kZXggNWM3NzI5MGVlYy4uMTExN2U5NWVk
ZSAxMDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUvYWNjZXNzL2hlYXBhbV94bG9nLmgKKysrIGIv
c3JjL2luY2x1ZGUvYWNjZXNzL2hlYXBhbV94bG9nLmgKQEAgLTM0NSw4ICszNDUsOSBAQCB0
eXBlZGVmIHN0cnVjdCB4bF9oZWFwX2ZyZWV6ZV9wYWdlCiAJVHJhbnNhY3Rpb25JZCBzbmFw
c2hvdENvbmZsaWN0SG9yaXpvbjsKIAl1aW50MTYJCW5wbGFuczsKIAotCS8qIEZSRUVaRSBQ
TEFOUyBGT0xMT1cgKi8KLQkvKiBPRkZTRVQgTlVNQkVSIEFSUkFZIEZPTExPV1MgKi8KKwkv
KgorCSAqIEluIHBheWxvYWQgb2YgYmxrIDAgOiBGUkVFWkUgUExBTlMgYW5kIE9GRlNFVCBO
VU1CRVIgQVJSQVkKKwkgKi8KIH0geGxfaGVhcF9mcmVlemVfcGFnZTsKIAogI2RlZmluZSBT
aXplT2ZIZWFwRnJlZXplUGFnZSAob2Zmc2V0b2YoeGxfaGVhcF9mcmVlemVfcGFnZSwgbnBs
YW5zKSArIHNpemVvZih1aW50MTYpKQpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvYWNjZXNz
L25idHhsb2cuaCBiL3NyYy9pbmNsdWRlL2FjY2Vzcy9uYnR4bG9nLmgKaW5kZXggM2IyZDk1
OWM2OS4uMWFhOGU3ZWNhNSAxMDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUvYWNjZXNzL25idHhs
b2cuaAorKysgYi9zcmMvaW5jbHVkZS9hY2Nlc3MvbmJ0eGxvZy5oCkBAIC0yMzYsOSArMjM2
LDEyIEBAIHR5cGVkZWYgc3RydWN0IHhsX2J0cmVlX2RlbGV0ZQogCXVpbnQxNgkJbmRlbGV0
ZWQ7CiAJdWludDE2CQludXBkYXRlZDsKIAotCS8qIERFTEVURUQgVEFSR0VUIE9GRlNFVCBO
VU1CRVJTIEZPTExPVyAqLwotCS8qIFVQREFURUQgVEFSR0VUIE9GRlNFVCBOVU1CRVJTIEZP
TExPVyAqLwotCS8qIFVQREFURUQgVFVQTEVTIE1FVEFEQVRBICh4bF9idHJlZV91cGRhdGUp
IEFSUkFZIEZPTExPV1MgKi8KKwkvKgorCSAqIEluIHBheWxvYWQgb2YgYmxrIDAgOgorCSAq
IC0gREVMRVRFRCBUQVJHRVQgT0ZGU0VUIE5VTUJFUlMKKwkgKiAtIFVQREFURUQgVEFSR0VU
IE9GRlNFVCBOVU1CRVJTCisJICogLSBVUERBVEVEIFRVUExFUyBNRVRBREFUQSAoeGxfYnRy
ZWVfdXBkYXRlKSBBUlJBWQorCSAqLwogfSB4bF9idHJlZV9kZWxldGU7CiAKICNkZWZpbmUg
U2l6ZU9mQnRyZWVEZWxldGUJKG9mZnNldG9mKHhsX2J0cmVlX2RlbGV0ZSwgbnVwZGF0ZWQp
ICsgc2l6ZW9mKHVpbnQxNikpCg==

--------------ekWnxcRaZg9zY3cB20IsJq2k--