Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rSLDc-00D87A-7L for pgsql-bugs@arkaria.postgresql.org; Tue, 23 Jan 2024 18:09:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rSLDb-00Eyl8-8M for pgsql-bugs@arkaria.postgresql.org; Tue, 23 Jan 2024 18:09:51 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rSLDa-00Eyl0-Np for pgsql-bugs@lists.postgresql.org; Tue, 23 Jan 2024 18:09:50 +0000 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rSLDX-003Kxs-Sw for pgsql-bugs@postgresql.org; Tue, 23 Jan 2024 18:09:50 +0000 Received: from mail-nwsmtp-smtp-corp-main-80.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-80.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:4da0:0:640:817e:0]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTPS id 8D4A462475; Tue, 23 Jan 2024 21:09:45 +0300 (MSK) Received: from smtpclient.apple (unknown [2a02:6b8:b081:8824::1:2]) by mail-nwsmtp-smtp-corp-main-80.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id i9Xw830IdSw0-oaplaydk; Tue, 23 Jan 2024 21:09:44 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1706033385; bh=0uSXh9nV6q5risTiCqRM3bFTPhpF5jtelAaaOqd5USI=; h=References:To:Cc:In-Reply-To:From:Date:Subject:Message-Id; b=Uqnlw6VLxzW9ICMNUzK4M3LcyUxgXUMSDEhhKCLquEIrFNkKh7c2IGTwWvu3H66mV aLJlK4I2yHq5A7OCt1cA8LlnpqtadxCgK8skyy2/A/JQFObbSeZjLqqSpmXyvZltL0 F+VzeHFkmfShea8ZzKtWkOEVhxh9v3L4EgiAcMEE= Authentication-Results: mail-nwsmtp-smtp-corp-main-80.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: "Andrey M. Borodin" Message-Id: Content-Type: multipart/mixed; boundary="Apple-Mail=_4A69F882-0D3A-433E-BC89-B6809419344C" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: [BUG] false positive in bt_index_check in case of short 4B varlena datum Date: Tue, 23 Jan 2024 23:09:43 +0500 In-Reply-To: Cc: Michael Zhilin , pgsql-bugs@postgresql.org, Yura Sokolov To: Alexander Lakhin References: <7bdbe559-d61a-4ae4-a6e1-48abdf3024cc@postgrespro.ru> <0FDE2089-D306-4CBB-AD1F-EC4B419E3B33@yandex-team.ru> <49BBA065-4C9A-4E50-9048-B457907FF219@yandex-team.ru> <76bc0dc9-4e43-4cd8-8eec-249b254ed1c9@postgrespro.ru> <8C83FCCA-2548-499A-8B1C-96C3D8ADB787@yandex-team.ru> X-Mailer: Apple Mail (2.3696.120.41.1.4) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --Apple-Mail=_4A69F882-0D3A-433E-BC89-B6809419344C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 20 Jan 2024, at 09:00, Alexander Lakhin = wrote: >=20 >> Alexander, do you plan to provide fixes for bugs you discovered? >=20 > No, I don't have a concrete proposal how to fix those bugs. I'd = thought > that fixing the whole class of such anomalies, not only one case, is a = good > thing to do, but if it's too complicated, maybe other similar bugs = could be > put aside. PFA draft fixes for both this errors. Alexander, Michael, Jian, what do = you think? I did not touch anything in first step - fix for original bug in this = thread. However, I think that comments from Jian He worth incorporating = into the fix. Best regards, Andrey Borodin. --Apple-Mail=_4A69F882-0D3A-433E-BC89-B6809419344C Content-Disposition: attachment; filename=v3-0001-contrib-amcheck-must-support-different-header-siz.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v3-0001-contrib-amcheck-must-support-different-header-siz.patch" Content-Transfer-Encoding: quoted-printable =46rom=20b514df4afa0c471cea90c84c3977cbc18c13fae4=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Michael=20Zhilin=20=0A= Date:=20Thu,=2014=20Dec=202023=2016:08:15=20+0300=0ASubject:=20[PATCH=20= v3=201/3]=20contrib/amcheck:=20must=20support=20different=20header=20= size=20of=0A=20short=20varlena=20datum=0A=0A---=0A=20= contrib/amcheck/expected/check_btree.out=20|=2019=20+++++++++++++=0A=20= contrib/amcheck/sql/check_btree.sql=20=20=20=20=20=20|=2017=20= +++++++++++=0A=20contrib/amcheck/verify_nbtree.c=20=20=20=20=20=20=20=20=20= =20|=2036=20++++++++++++++++++++----=0A=203=20files=20changed,=2067=20= insertions(+),=205=20deletions(-)=0A=0Adiff=20--git=20= a/contrib/amcheck/expected/check_btree.out=20= b/contrib/amcheck/expected/check_btree.out=0Aindex=20= 86b38d93f4..e37901c402=20100644=0A---=20= a/contrib/amcheck/expected/check_btree.out=0A+++=20= b/contrib/amcheck/expected/check_btree.out=0A@@=20-1,3=20+1,5=20@@=0A+--=20= directory=20paths=20are=20passed=20to=20us=20in=20environment=20= variables=0A+\getenv=20abs_builddir=20PG_ABS_BUILDDIR=0A=20CREATE=20= TABLE=20bttest_a(id=20int8);=0A=20CREATE=20TABLE=20bttest_b(id=20int8);=0A= =20CREATE=20TABLE=20bttest_multi(id=20int8,=20data=20int8);=0A@@=20= -240,6=20+242,22=20@@=20SELECT=20= bt_index_check('bttest_unique_nulls_b_c_idx',=20heapallindexed=20=3D>=20= true,=20che=0A=20=20=0A=20(1=20row)=0A=20=0A+--=0A+--=20BUG:=20must=20= support=20different=20header=20size=20of=20short=20varlena=20datum=0A+--=0A= +CREATE=20TABLE=20varlena_bug=20(v=20text);=0A+ALTER=20TABLE=20= varlena_bug=20ALTER=20column=20v=20SET=20storage=20plain;=0A+INSERT=20= INTO=20varlena_bug=20VALUES=20('x');=0A+\set=20filename=20:abs_builddir=20= '/results/varlena_bug.dmp'=0A+COPY=20varlena_bug=20TO=20:'filename';=0A= +COPY=20varlena_bug=20FROM=20:'filename';=0A+CREATE=20INDEX=20= varlena_bug_idx=20on=20varlena_bug(v);=0A+SELECT=20= bt_index_check('varlena_bug_idx',=20true);=0A+=20bt_index_check=20=0A= +----------------=0A+=20=0A+(1=20row)=0A+=0A=20--=20cleanup=0A=20DROP=20= TABLE=20bttest_a;=0A=20DROP=20TABLE=20bttest_b;=0A@@=20-250,3=20+268,4=20= @@=20DROP=20FUNCTION=20ifun(int8);=0A=20DROP=20TABLE=20= bttest_unique_nulls;=0A=20DROP=20OWNED=20BY=20regress_bttest_role;=20--=20= permissions=0A=20DROP=20ROLE=20regress_bttest_role;=0A+DROP=20TABLE=20= varlena_bug;=0Adiff=20--git=20a/contrib/amcheck/sql/check_btree.sql=20= b/contrib/amcheck/sql/check_btree.sql=0Aindex=20aa461f7fb9..e67eb6352f=20= 100644=0A---=20a/contrib/amcheck/sql/check_btree.sql=0A+++=20= b/contrib/amcheck/sql/check_btree.sql=0A@@=20-1,3=20+1,6=20@@=0A+--=20= directory=20paths=20are=20passed=20to=20us=20in=20environment=20= variables=0A+\getenv=20abs_builddir=20PG_ABS_BUILDDIR=0A+=0A=20CREATE=20= TABLE=20bttest_a(id=20int8);=0A=20CREATE=20TABLE=20bttest_b(id=20int8);=0A= =20CREATE=20TABLE=20bttest_multi(id=20int8,=20data=20int8);=0A@@=20= -148,6=20+151,19=20@@=20SELECT=20= bt_index_check('bttest_unique_nulls_c_key',=20heapallindexed=20=3D>=20= true,=20check=0A=20CREATE=20INDEX=20on=20bttest_unique_nulls=20(b,c);=0A=20= SELECT=20bt_index_check('bttest_unique_nulls_b_c_idx',=20heapallindexed=20= =3D>=20true,=20checkunique=20=3D>=20true);=0A=20=0A+--=0A+--=20BUG:=20= must=20support=20different=20header=20size=20of=20short=20varlena=20= datum=0A+--=0A+=0A+CREATE=20TABLE=20varlena_bug=20(v=20text);=0A+ALTER=20= TABLE=20varlena_bug=20ALTER=20column=20v=20SET=20storage=20plain;=0A= +INSERT=20INTO=20varlena_bug=20VALUES=20('x');=0A+\set=20filename=20= :abs_builddir=20'/results/varlena_bug.dmp'=0A+COPY=20varlena_bug=20TO=20= :'filename';=0A+COPY=20varlena_bug=20FROM=20:'filename';=0A+CREATE=20= INDEX=20varlena_bug_idx=20on=20varlena_bug(v);=0A+SELECT=20= bt_index_check('varlena_bug_idx',=20true);=0A+=0A=20--=20cleanup=0A=20= DROP=20TABLE=20bttest_a;=0A=20DROP=20TABLE=20bttest_b;=0A@@=20-158,3=20= +174,4=20@@=20DROP=20FUNCTION=20ifun(int8);=0A=20DROP=20TABLE=20= bttest_unique_nulls;=0A=20DROP=20OWNED=20BY=20regress_bttest_role;=20--=20= permissions=0A=20DROP=20ROLE=20regress_bttest_role;=0A+DROP=20TABLE=20= varlena_bug;=0Adiff=20--git=20a/contrib/amcheck/verify_nbtree.c=20= b/contrib/amcheck/verify_nbtree.c=0Aindex=2091caa53dd8..e7f01c2add=20= 100644=0A---=20a/contrib/amcheck/verify_nbtree.c=0A+++=20= b/contrib/amcheck/verify_nbtree.c=0A@@=20-2942,7=20+2942,7=20@@=20= bt_normalize_tuple(BtreeCheckState=20*state,=20IndexTuple=20itup)=0A=20=09= TupleDesc=09tupleDescriptor=20=3D=20RelationGetDescr(state->rel);=0A=20=09= Datum=09=09normalized[INDEX_MAX_KEYS];=0A=20=09bool=09=09= isnull[INDEX_MAX_KEYS];=0A-=09bool=09=09toast_free[INDEX_MAX_KEYS];=0A+=09= bool=09=09need_free[INDEX_MAX_KEYS];=0A=20=09bool=09=09formnewtup=20=3D=20= false;=0A=20=09IndexTuple=09reformed;=0A=20=09int=09=09=09i;=0A@@=20= -2961,7=20+2961,7=20@@=20bt_normalize_tuple(BtreeCheckState=20*state,=20= IndexTuple=20itup)=0A=20=09=09att=20=3D=20TupleDescAttr(tupleDescriptor,=20= i);=0A=20=0A=20=09=09/*=20Assume=20untoasted/already=20normalized=20= datum=20initially=20*/=0A-=09=09toast_free[i]=20=3D=20false;=0A+=09=09= need_free[i]=20=3D=20false;=0A=20=09=09normalized[i]=20=3D=20= index_getattr(itup,=20att->attnum,=0A=20=09=09=09=09=09=09=09=09=09=20=20= tupleDescriptor,=0A=20=09=09=09=09=09=09=09=09=09=20=20&isnull[i]);=0A@@=20= -2973,6=20+2973,7=20@@=20bt_normalize_tuple(BtreeCheckState=20*state,=20= IndexTuple=20itup)=0A=20=09=09=20*=20index=20without=20further=20= processing,=20so=20an=20external=20varlena=20header=0A=20=09=09=20*=20= should=20never=20be=20encountered=20here=0A=20=09=09=20*/=0A+=0A=20=09=09= if=20(VARATT_IS_EXTERNAL(DatumGetPointer(normalized[i])))=0A=20=09=09=09= ereport(ERROR,=0A=20=09=09=09=09=09(errcode(ERRCODE_INDEX_CORRUPTED),=0A= @@=20-2984,11=20+2985,32=20@@=20bt_normalize_tuple(BtreeCheckState=20= *state,=20IndexTuple=20itup)=0A=20=09=09{=0A=20=09=09=09formnewtup=20=3D=20= true;=0A=20=09=09=09normalized[i]=20=3D=20= PointerGetDatum(PG_DETOAST_DATUM(normalized[i]));=0A-=09=09=09= toast_free[i]=20=3D=20true;=0A+=09=09=09need_free[i]=20=3D=20true;=0A=20=09= =09}=0A+=09=09/*=0A+=09=09=20*=20Short=20tuples=20may=20have=201B=20or=20= 4B=20header.=20Convert=204B=20header=20of=20short=0A+=09=09=20*=20tuples=20= to=201B=0A+=09=09=20*/=0A+=09=09else=20if=20= (VARATT_CAN_MAKE_SHORT(DatumGetPointer(normalized[i])))=0A+=09=09{=0A+=09= =09=09/*=20convert=20to=20short=20varlena=20*/=0A+=09=09=09Size=20len=20= =3D=20VARATT_CONVERTED_SHORT_SIZE(DatumGetPointer(normalized[i]));=0A+=09= =09=09char=20*data=20=3D=20palloc(len);=0A+=0A+=09=09=09= SET_VARSIZE_SHORT(data,=20len);=0A+=09=09=09memcpy(data=20+=201,=20= VARDATA(DatumGetPointer(normalized[i])),=20len=20-=201);=0A+=0A+=09=09=09= formnewtup=20=3D=20true;=0A+=09=09=09normalized[i]=20=3D=20= PointerGetDatum(data);=0A+=09=09=09need_free[i]=20=3D=20true;=0A+=09=09}=0A= +=0A=20=09}=0A=20=0A-=09/*=20Easier=20case:=20Tuple=20has=20varlena=20= datums,=20none=20of=20which=20are=20compressed=20*/=0A+=09/*=0A+=09=20*=20= Easier=20case:=20Tuple=20has=20varlena=20datums,=20none=20of=20which=20= are=20compressed=20or=0A+=09=20*=20short=20with=204B=20header=0A+=09=20= */=0A=20=09if=20(!formnewtup)=0A=20=09=09return=20itup;=0A=20=0A@@=20= -2997,6=20+3019,10=20@@=20bt_normalize_tuple(BtreeCheckState=20*state,=20= IndexTuple=20itup)=0A=20=09=20*=20creating=20normalized=20version=20of=20= the=20tuple=20from=20uncompressed=20input=20datums=0A=20=09=20*=20= (normalized=20input=20datums).=20=20This=20is=20rather=20naive,=20but=20= shouldn't=20be=0A=20=09=20*=20necessary=20too=20often.=0A+=09=20*=20Also=20= tuple=20had=20short=20varlena=20datums=20with=204B=20header.=20Actually=20= there=20is=20no=0A+=09=20*=20restriction=20with=20have=20heap=20tuple=20= containing=20varlena=20datum=20with=204B=20header=0A+=09=20*=20and=20= corresponding=20index=20tuple=20containing=20varlena=20datum=20with=201B=20= header.=0A+=09=20*=20For=20fingerprinting=20let's=20convert=20heap=20= tuple=20varlena=20datum=20to=201B=20format.=0A=20=09=20*=0A=20=09=20*=20= Note=20that=20we=20rely=20on=20deterministic=20index_form_tuple()=20= TOAST=20compression=0A=20=09=20*=20of=20normalized=20input.=0A@@=20= -3006,7=20+3032,7=20@@=20bt_normalize_tuple(BtreeCheckState=20*state,=20= IndexTuple=20itup)=0A=20=0A=20=09/*=20Cannot=20leak=20memory=20here=20*/=0A= =20=09for=20(i=20=3D=200;=20i=20<=20tupleDescriptor->natts;=20i++)=0A-=09= =09if=20(toast_free[i])=0A+=09=09if=20(need_free[i])=0A=20=09=09=09= pfree(DatumGetPointer(normalized[i]));=0A=20=0A=20=09return=20reformed;=0A= --=20=0A2.37.1=20(Apple=20Git-137.1)=0A=0A= --Apple-Mail=_4A69F882-0D3A-433E-BC89-B6809419344C Content-Disposition: attachment; filename=v3-0003-amcheck-avoid-failing-on-oversized-tuples.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v3-0003-amcheck-avoid-failing-on-oversized-tuples.patch" Content-Transfer-Encoding: quoted-printable =46rom=2001dd6155276ed97adcad3e5c55c064b7b8116bfd=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20"Andrey=20M.=20Borodin"=20=0A= Date:=20Tue,=2023=20Jan=202024=2023:03:28=20+0500=0ASubject:=20[PATCH=20= v3=203/3]=20amcheck:=20avoid=20failing=20on=20oversized=20tuples=0A=0A= Due=20to=20changes=20in=20toast=20policies,=20some=20heap=20tuples=20= might=20become=20too=0Abig=20index=20tuples.=20This=20commit=20prevents=20= ERRORs=20in=20this=20case,=20because=0Athis=20anomaly=20does=20not=20= create=20dangerous=20conditions=20to=20DMS.=20However,=0Athe=20database=20= cannot=20be=20dumped-restored,=20so=20we=20emit=20a=20NOTICE.=0A=0A= Reported-by:=20Alexander=20Lakhin=0A---=0A=20= contrib/amcheck/expected/check_btree.out=20|=2014=20++++++=0A=20= contrib/amcheck/sql/check_btree.sql=20=20=20=20=20=20|=2011=20+++++=0A=20= contrib/amcheck/verify_nbtree.c=20=20=20=20=20=20=20=20=20=20|=2054=20= +++++++++++++++++++++---=0A=203=20files=20changed,=2073=20insertions(+),=20= 6=20deletions(-)=0A=0Adiff=20--git=20= a/contrib/amcheck/expected/check_btree.out=20= b/contrib/amcheck/expected/check_btree.out=0Aindex=20= 36b1ebe34c..91394bed2b=20100644=0A---=20= a/contrib/amcheck/expected/check_btree.out=0A+++=20= b/contrib/amcheck/expected/check_btree.out=0A@@=20-270,6=20+270,20=20@@=20= SELECT=20bt_index_check('tbl_idx',=20true);=0A=20=20=0A=20(1=20row)=0A=20= =0A+DROP=20TABLE=20tbl;=0A+--=20Check=20oversized=20datums=20that=20= cannot=20be=20inserted=20into=20index=0A+CREATE=20TABLE=20t(f1=20text);=0A= +CREATE=20INDEX=20t_idx=20ON=20t(f1);=0A+INSERT=20INTO=20t=20= VALUES(repeat('1234567890',=201000));=0A+ALTER=20TABLE=20t=20ALTER=20= COLUMN=20f1=20SET=20STORAGE=20plain;=0A+SELECT=20bt_index_check('t_idx',=20= true);=0A+NOTICE:=20=20Index=20contain=20tuples=20that=20cannot=20fit=20= into=20index=20page,=20if=20toasted=20with=20current=20toast=20policy=0A= +=20bt_index_check=20=0A+----------------=0A+=20=0A+(1=20row)=0A+=0A= +DROP=20TABLE=20t;=0A=20--=20cleanup=0A=20DROP=20TABLE=20bttest_a;=0A=20= DROP=20TABLE=20bttest_b;=0Adiff=20--git=20= a/contrib/amcheck/sql/check_btree.sql=20= b/contrib/amcheck/sql/check_btree.sql=0Aindex=200a827edd1a..cdfbc8a141=20= 100644=0A---=20a/contrib/amcheck/sql/check_btree.sql=0A+++=20= b/contrib/amcheck/sql/check_btree.sql=0A@@=20-172,6=20+172,17=20@@=20= INSERT=20INTO=20tbl=20VALUES=20(repeat('Test',=20250));=0A=20ALTER=20= TABLE=20tbl=20ALTER=20COLUMN=20t=20SET=20STORAGE=20extended;=0A=20=0A=20= SELECT=20bt_index_check('tbl_idx',=20true);=0A+DROP=20TABLE=20tbl;=0A+=0A= +--=20Check=20oversized=20datums=20that=20cannot=20be=20inserted=20into=20= index=0A+=0A+CREATE=20TABLE=20t(f1=20text);=0A+CREATE=20INDEX=20t_idx=20= ON=20t(f1);=0A+INSERT=20INTO=20t=20VALUES(repeat('1234567890',=201000));=0A= +ALTER=20TABLE=20t=20ALTER=20COLUMN=20f1=20SET=20STORAGE=20plain;=0A+=0A= +SELECT=20bt_index_check('t_idx',=20true);=0A+DROP=20TABLE=20t;=0A=20=0A=20= --=20cleanup=0A=20DROP=20TABLE=20bttest_a;=0Adiff=20--git=20= a/contrib/amcheck/verify_nbtree.c=20b/contrib/amcheck/verify_nbtree.c=0A= index=206654b5afe7..f70e01191b=20100644=0A---=20= a/contrib/amcheck/verify_nbtree.c=0A+++=20= b/contrib/amcheck/verify_nbtree.c=0A@@=20-128,6=20+128,11=20@@=20typedef=20= struct=20BtreeCheckState=0A=20=09bloom_filter=20*filter;=0A=20=09/*=20= Debug=20counter=20*/=0A=20=09int64=09=09heaptuplespresent;=0A+=09/*=0A+=09= =20*=20During=20check=20we=20might=20find=20tuples=20that=20due=20to=20= current=20TOAST=20policies=0A+=09=20*=20should=20not=20reside=20in=20= index,=20but=20still=20are=20there.=0A+=09=20*/=0A+=09bool=20= has_oversized_tuples;=0A=20}=20BtreeCheckState;=0A=20=0A=20/*=0A@@=20= -1624,10=20+1629,17=20@@=20bt_target_page_check(BtreeCheckState=20= *state)=0A=20=0A=20=09=09=09=09=09logtuple=20=3D=20= bt_posting_plain_tuple(itup,=20i);=0A=20=09=09=09=09=09norm=20=3D=20= bt_normalize_tuple(state,=20logtuple);=0A-=09=09=09=09=09= bloom_add_element(state->filter,=20(unsigned=20char=20*)=20norm,=0A+=09=09= =09=09=09if=20(norm=20=3D=3D=20NULL)=0A+=09=09=09=09=09{=0A+=09=09=09=09=09= =09if=20(!state->has_oversized_tuples)=0A+=09=09=09=09=09=09=09= elog(NOTICE,=20"Index=20contain=20tuples=20that=20cannot=20fit=20into=20= index=20page,=20if=20toasted=20with=20current=20toast=20policy");=0A+=09=09= =09=09=09=09state->has_oversized_tuples=20=3D=20true;=0A+=09=09=09=09=09= }=0A+=09=09=09=09=09else=0A+=09=09=09=09=09=09= bloom_add_element(state->filter,=20(unsigned=20char=20*)=20norm,=0A=20=09= =09=09=09=09=09=09=09=09=20=20IndexTupleSize(norm));=0A=20=09=09=09=09=09= /*=20Be=20tidy=20*/=0A-=09=09=09=09=09if=20(norm=20!=3D=20logtuple)=0A+=09= =09=09=09=09if=20(norm=20!=3D=20logtuple=20&&=20norm=20!=3D=20NULL)=0A=20= =09=09=09=09=09=09pfree(norm);=0A=20=09=09=09=09=09pfree(logtuple);=0A=20= =09=09=09=09}=0A@@=20-1635,10=20+1647,17=20@@=20= bt_target_page_check(BtreeCheckState=20*state)=0A=20=09=09=09else=0A=20=09= =09=09{=0A=20=09=09=09=09norm=20=3D=20bt_normalize_tuple(state,=20itup);=0A= -=09=09=09=09bloom_add_element(state->filter,=20(unsigned=20char=20*)=20= norm,=0A-=09=09=09=09=09=09=09=09=20=20IndexTupleSize(norm));=0A+=09=09=09= =09if=20(norm=20=3D=3D=20NULL)=0A+=09=09=09=09{=0A+=09=09=09=09=09if=20= (!state->has_oversized_tuples)=0A+=09=09=09=09=09=09elog(NOTICE,=20= "Index=20contain=20tuples=20that=20cannot=20fit=20into=20index=20page,=20= if=20toasted=20with=20current=20toast=20policy");=0A+=09=09=09=09=09= state->has_oversized_tuples=20=3D=20true;=0A+=09=09=09=09}=0A+=09=09=09=09= else=0A+=09=09=09=09=09bloom_add_element(state->filter,=20(unsigned=20= char=20*)=20norm,=0A+=09=09=09=09=09=09=09=09IndexTupleSize(norm));=0A=20= =09=09=09=09/*=20Be=20tidy=20*/=0A-=09=09=09=09if=20(norm=20!=3D=20itup)=0A= +=09=09=09=09if=20(norm=20!=3D=20itup=20&&=20norm=20!=3D=20NULL)=0A=20=09= =09=09=09=09pfree(norm);=0A=20=09=09=09}=0A=20=09=09}=0A@@=20-2876,6=20= +2895,7=20@@=20bt_tuple_present_callback(Relation=20index,=20ItemPointer=20= tid,=20Datum=20*values,=0A=20=09BtreeCheckState=20*state=20=3D=20= (BtreeCheckState=20*)=20checkstate;=0A=20=09IndexTuple=09itup,=0A=20=09=09= =09=09norm;=0A+=09bool=20miss_oversized_tuple=20=3D=20false;=0A=20=0A=20=09= Assert(state->heapallindexed);=0A=20=0A@@=20-2883,9=20+2903,19=20@@=20= bt_tuple_present_callback(Relation=20index,=20ItemPointer=20tid,=20Datum=20= *values,=0A=20=09itup=20=3D=20index_form_tuple(RelationGetDescr(index),=20= values,=20isnull);=0A=20=09itup->t_tid=20=3D=20*tid;=0A=20=09norm=20=3D=20= bt_normalize_tuple(state,=20itup);=0A+=09if=20(norm=20=3D=3D=20NULL)=0A+=09= {=0A+=09=09if=20(state->has_oversized_tuples)=0A+=09=09{=0A+=09=09=09/*=20= exempt=20this=20oversized=20tuple=20*/=0A+=09=09=09= state->heaptuplespresent++;=0A+=09=09=09pfree(itup);=0A+=09=09=09return;=0A= +=09=09}=0A+=09}=0A=20=0A=20=09/*=20Probe=20Bloom=20filter=20--=20tuple=20= should=20be=20present=20*/=0A-=09if=20= (bloom_lacks_element(state->filter,=20(unsigned=20char=20*)=20norm,=0A+=09= if=20((norm=20=3D=3D=20NULL)=20||=20bloom_lacks_element(state->filter,=20= (unsigned=20char=20*)=20norm,=0A=20=09=09=09=09=09=09=09= IndexTupleSize(norm)))=0A=20=09=09ereport(ERROR,=0A=20=09=09=09=09= (errcode(ERRCODE_DATA_CORRUPTED),=0A@@=20-2936,6=20+2966,9=20@@=20= bt_tuple_present_callback(Relation=20index,=20ItemPointer=20tid,=20Datum=20= *values,=0A=20=20*=20Caller=20does=20normalization=20for=20non-pivot=20= tuples=20that=20have=20a=20posting=20list,=0A=20=20*=20since=20dummy=20= CREATE=20INDEX=20callback=20code=20generates=20new=20tuples=20with=20the=20= same=0A=20=20*=20normalized=20representation.=0A+=20*=20=0A+=20*=20If=20= the=20tuple=20is=20exampt=20from=20checking=20due=20to=20= has_oversized_tuples=20this=20function=0A+=20*=20returns=20NULL.=0A=20=20= */=0A=20static=20IndexTuple=0A=20bt_normalize_tuple(BtreeCheckState=20= *state,=20IndexTuple=20itup)=0A@@=20-2947,6=20+2980,7=20@@=20= bt_normalize_tuple(BtreeCheckState=20*state,=20IndexTuple=20itup)=0A=20=09= bool=09=09formnewtup=20=3D=20false;=0A=20=09IndexTuple=09reformed;=0A=20=09= int=09=09=09i;=0A+=09Size=09=09data_size;=0A=20=0A=20=09/*=20Caller=20= should=20only=20pass=20"logical"=20non-pivot=20tuples=20here=20*/=0A=20=09= Assert(!BTreeTupleIsPosting(itup)=20&&=20!BTreeTupleIsPivot(itup));=0A@@=20= -3026,6=20+3060,14=20@@=20bt_normalize_tuple(BtreeCheckState=20*state,=20= IndexTuple=20itup)=0A=20=09if=20(!formnewtup)=0A=20=09=09return=20itup;=0A= =20=0A+=09data_size=20=3D=20= MAXALIGN(heap_compute_data_size(tupleDescriptor,=0A+=09=09=09=09=09=09=09= =09=09=20=20=20normalized,=20isnull)=0A+=09=09=09=09+=20= MAXALIGN(sizeof(IndexTupleData)=20+=20= sizeof(IndexAttributeBitMapData)));=0A+=09if=20((data_size=20&=20= INDEX_SIZE_MASK)=20!=3D=20data_size)=0A+=09{=0A+=09=09return=20NULL;=0A+=09= }=0A+=0A=20=09/*=0A=20=09=20*=20Hard=20case:=20Tuple=20had=20compressed=20= varlena=20datums=20that=20necessitate=0A=20=09=20*=20creating=20= normalized=20version=20of=20the=20tuple=20from=20uncompressed=20input=20= datums=0A--=20=0A2.37.1=20(Apple=20Git-137.1)=0A=0A= --Apple-Mail=_4A69F882-0D3A-433E-BC89-B6809419344C Content-Disposition: attachment; filename=v3-0002-amcheck-prevent-false-positives-from-extended-dat.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v3-0002-amcheck-prevent-false-positives-from-extended-dat.patch" Content-Transfer-Encoding: quoted-printable =46rom=20e2ab4c0d3ad9777f70fccb21dd446e248cd5f52a=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20"Andrey=20M.=20Borodin"=20=0A= Date:=20Tue,=2023=20Jan=202024=2021:57:02=20+0500=0ASubject:=20[PATCH=20= v3=202/3]=20amcheck:=20prevent=20false=20positives=20from=20extended=20= datums=0A=0AReported-by:=20Alexander=20Lakhin=0A---=0A=20= contrib/amcheck/expected/check_btree.out=20|=2012=20++++++++++++=0A=20= contrib/amcheck/sql/check_btree.sql=20=20=20=20=20=20|=20=209=20= +++++++++=0A=20contrib/amcheck/verify_nbtree.c=20=20=20=20=20=20=20=20=20= =20|=2012=20++++++++++++=0A=203=20files=20changed,=2033=20insertions(+)=0A= =0Adiff=20--git=20a/contrib/amcheck/expected/check_btree.out=20= b/contrib/amcheck/expected/check_btree.out=0Aindex=20= e37901c402..36b1ebe34c=20100644=0A---=20= a/contrib/amcheck/expected/check_btree.out=0A+++=20= b/contrib/amcheck/expected/check_btree.out=0A@@=20-258,6=20+258,18=20@@=20= SELECT=20bt_index_check('varlena_bug_idx',=20true);=0A=20=20=0A=20(1=20= row)=0A=20=0A+--=20Check=20extended=20varlena=0A+CREATE=20TABLE=20tbl(t=20= text);=0A+ALTER=20TABLE=20tbl=20ALTER=20COLUMN=20t=20SET=20STORAGE=20= plain;=0A+CREATE=20INDEX=20tbl_idx=20ON=20tbl=20(t);=0A+INSERT=20INTO=20= tbl=20VALUES=20(repeat('Test',=20250));=0A+ALTER=20TABLE=20tbl=20ALTER=20= COLUMN=20t=20SET=20STORAGE=20extended;=0A+SELECT=20= bt_index_check('tbl_idx',=20true);=0A+=20bt_index_check=20=0A= +----------------=0A+=20=0A+(1=20row)=0A+=0A=20--=20cleanup=0A=20DROP=20= TABLE=20bttest_a;=0A=20DROP=20TABLE=20bttest_b;=0Adiff=20--git=20= a/contrib/amcheck/sql/check_btree.sql=20= b/contrib/amcheck/sql/check_btree.sql=0Aindex=20e67eb6352f..0a827edd1a=20= 100644=0A---=20a/contrib/amcheck/sql/check_btree.sql=0A+++=20= b/contrib/amcheck/sql/check_btree.sql=0A@@=20-164,6=20+164,15=20@@=20= COPY=20varlena_bug=20FROM=20:'filename';=0A=20CREATE=20INDEX=20= varlena_bug_idx=20on=20varlena_bug(v);=0A=20SELECT=20= bt_index_check('varlena_bug_idx',=20true);=0A=20=0A+--=20Check=20= extended=20varlena=0A+CREATE=20TABLE=20tbl(t=20text);=0A+ALTER=20TABLE=20= tbl=20ALTER=20COLUMN=20t=20SET=20STORAGE=20plain;=0A+CREATE=20INDEX=20= tbl_idx=20ON=20tbl=20(t);=0A+INSERT=20INTO=20tbl=20VALUES=20= (repeat('Test',=20250));=0A+ALTER=20TABLE=20tbl=20ALTER=20COLUMN=20t=20= SET=20STORAGE=20extended;=0A+=0A+SELECT=20bt_index_check('tbl_idx',=20= true);=0A+=0A=20--=20cleanup=0A=20DROP=20TABLE=20bttest_a;=0A=20DROP=20= TABLE=20bttest_b;=0Adiff=20--git=20a/contrib/amcheck/verify_nbtree.c=20= b/contrib/amcheck/verify_nbtree.c=0Aindex=20e7f01c2add..6654b5afe7=20= 100644=0A---=20a/contrib/amcheck/verify_nbtree.c=0A+++=20= b/contrib/amcheck/verify_nbtree.c=0A@@=20-23,6=20+23,7=20@@=0A=20=20*/=0A= =20#include=20"postgres.h"=0A=20=0A+#include=20"access/heaptoast.h"=0A=20= #include=20"access/htup_details.h"=0A=20#include=20"access/nbtree.h"=0A=20= #include=20"access/table.h"=0A@@=20-2981,6=20+2982,17=20@@=20= bt_normalize_tuple(BtreeCheckState=20*state,=20IndexTuple=20itup)=0A=20=09= =09=09=09=09=09=09ItemPointerGetBlockNumber(&(itup->t_tid)),=0A=20=09=09=09= =09=09=09=09ItemPointerGetOffsetNumber(&(itup->t_tid)),=0A=20=09=09=09=09= =09=09=09RelationGetRelationName(state->rel))));=0A+=09=09else=20if=20= (!VARATT_IS_EXTENDED(DatumGetPointer(normalized[i]))=20&&=0A+=09=09=09= VARSIZE(DatumGetPointer(normalized[i]))=20>=20TOAST_INDEX_TARGET=20&&=0A= +=09=09=09(att->attstorage=20=3D=3D=20TYPSTORAGE_EXTENDED=20||=0A+=09=09=09= =20att->attstorage=20=3D=3D=20TYPSTORAGE_MAIN))=0A+=09=09{=0A+=09=09=09= /*=0A+=09=09=09=20*=20this=20attribute=20will=20be=20compressed=20by=20= index_form_tuple(),=0A+=09=09=09=20*=20this=20might=20be=20already=20= done=20in=20heap,=20so=20force=20forming.=0A+=09=09=09=20*/=0A+=09=09=09= formnewtup=20=3D=20true;=0A+=09=09}=0A=20=09=09else=20if=20= (VARATT_IS_COMPRESSED(DatumGetPointer(normalized[i])))=0A=20=09=09{=0A=20= =09=09=09formnewtup=20=3D=20true;=0A--=20=0A2.37.1=20(Apple=20Git-137.1)=0A= =0A= --Apple-Mail=_4A69F882-0D3A-433E-BC89-B6809419344C--