Content-Type: multipart/mixed; boundary="------------s8TMrgCqChb4s73L1OMfgQVt"
Message-ID: <7bdbe559-d61a-4ae4-a6e1-48abdf3024cc@postgrespro.ru>
Date: Thu, 14 Dec 2023 19:18:11 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
To: pgsql-bugs@postgresql.org
Cc: y sokolov <y.sokolov@postgrespro.ru>
From: Michael Zhilin <m.zhilin@postgrespro.ru>
Subject: [BUG] false positive in bt_index_check in case of short 4B varlena
 datum
Archived-At: 
 <https://www.postgresql.org/message-id/7bdbe559-d61a-4ae4-a6e1-48abdf3024cc%40postgrespro.ru>
Precedence: bulk

This is a multi-part message in MIME format.
--------------s8TMrgCqChb4s73L1OMfgQVt
Content-Type: multipart/alternative;
 boundary="------------HUKiOIvzzlKguTDiFe5oQ3Oc"

--------------HUKiOIvzzlKguTDiFe5oQ3Oc
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

Following example produces error raised from bt_index_check.

drop table if exists t;
create table t (v text);
alter table t alter column v set storage plain;
insert into t values ('x');
copy t to '/tmp/1.lst';
copy t from '/tmp/1.lst';
create index t_idx on t(v);
create extension if not exists amcheck;
select bt_index_check('t_idx', true);

postgres=# select bt_index_check('t_idx', true);
ERROR:  heap tuple (0,2) from table "t" lacks matching index tuple 
within index "t_idx"
HINT:  Retrying verification using the function bt_index_parent_check() 
might provide a more specific error.

As result table contains 2 logically identical tuples:
  - one contains varlena 'x' with 1B (1-byte) header (added by INSERT 
statement)
  - one contains varlena 'x' with 4B (4-bytes) header (added by COPY 
statement)
CREATE INDEX statement builds index with posting list referencing both 
heap tuples.
The function bt_index_check calculates fingerprints of 1B and 4B header 
datums,
they are different and function returns error.

The attached patch allows to avoid such kind of false positives by 
converting short
4B datums to 1B before fingerprinting. Also it contains test for 
provided case.

Thank you,
  Michael

-- 
Michael Zhilin
Postgres Professional
--------------HUKiOIvzzlKguTDiFe5oQ3Oc
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <font face="monospace">Hi,<br>
      <br>
      Following example produces error raised from bt_index_check.<br>
      <br>
      drop table if exists t;<br>
      create table t (v text);<br>
      alter table t alter column v set storage plain;<br>
      insert into t values ('x');<br>
      copy t to '/tmp/1.lst';<br>
      copy t from '/tmp/1.lst';<br>
      create index t_idx on t(v);<br>
      create extension if not exists amcheck;<br>
      select bt_index_check('t_idx', true);<br>
      <br>
      postgres=# select bt_index_check('t_idx', true);<br>
      ERROR:  heap tuple (0,2) from table "t" lacks matching index tuple
      within index "t_idx"<br>
      HINT:  Retrying verification using the function
      bt_index_parent_check() might provide a more specific error.<br>
      <br>
      As result table contains 2 logically identical tuples:<br>
       - one contains varlena 'x' with 1B (1-byte) header (added by
      INSERT statement)<br>
       - one contains varlena 'x' with 4B (4-bytes) header (added by
      COPY statement)<br>
      CREATE INDEX statement builds index with posting list referencing
      both heap tuples.<br>
      The function bt_index_check calculates fingerprints of 1B and 4B
      header datums, <br>
      they are different and function returns error.<br>
      <br>
      The attached patch allows to avoid such kind of false positives by
      converting short <br>
      4B datums to 1B before fingerprinting. Also it contains test for
      provided case.<br>
      <br>
      Thank you,<br>
       Michael<br>
      <br>
      -- <br>
      Michael Zhilin<br>
      Postgres Professional<br>
    </font>
  </body>
</html>

--------------HUKiOIvzzlKguTDiFe5oQ3Oc--

--------------s8TMrgCqChb4s73L1OMfgQVt
Content-Type: text/x-patch; charset=UTF-8;
 name="0001-contrib-amcheck-must-support-different-header-size-o.patch"
Content-Disposition: attachment;
 filename*0="0001-contrib-amcheck-must-support-different-header-size-o.pa";
 filename*1="tch"
Content-Transfer-Encoding: base64

RnJvbSBlMmYxNTI4MTA3ZTJhMzlhN2YzYzAxYmRmYzc0NTJhOWZlMTdlZWVlIE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBNaWNoYWVsIFpoaWxpbiA8bWl6aGthQEZyZWVCU0Qu
b3JnPgpEYXRlOiBUaHUsIDE0IERlYyAyMDIzIDE2OjA4OjE1ICswMzAwClN1YmplY3Q6IFtQ
QVRDSF0gY29udHJpYi9hbWNoZWNrOiBtdXN0IHN1cHBvcnQgZGlmZmVyZW50IGhlYWRlciBz
aXplIG9mIHNob3J0CiB2YXJsZW5hIGRhdHVtCgotLS0KIGNvbnRyaWIvYW1jaGVjay9leHBl
Y3RlZC9jaGVja19idHJlZS5vdXQgfCAxOSArKysrKysrKysrKysrCiBjb250cmliL2FtY2hl
Y2svc3FsL2NoZWNrX2J0cmVlLnNxbCAgICAgIHwgMTcgKysrKysrKysrKysKIGNvbnRyaWIv
YW1jaGVjay92ZXJpZnlfbmJ0cmVlLmMgICAgICAgICAgfCAzNiArKysrKysrKysrKysrKysr
KysrKy0tLS0KIDMgZmlsZXMgY2hhbmdlZCwgNjcgaW5zZXJ0aW9ucygrKSwgNSBkZWxldGlv
bnMoLSkKCmRpZmYgLS1naXQgYS9jb250cmliL2FtY2hlY2svZXhwZWN0ZWQvY2hlY2tfYnRy
ZWUub3V0IGIvY29udHJpYi9hbWNoZWNrL2V4cGVjdGVkL2NoZWNrX2J0cmVlLm91dAppbmRl
eCAzODc5MWJiYzFmNC4uZGI3M2Y4ZmQ1YjMgMTAwNjQ0Ci0tLSBhL2NvbnRyaWIvYW1jaGVj
ay9leHBlY3RlZC9jaGVja19idHJlZS5vdXQKKysrIGIvY29udHJpYi9hbWNoZWNrL2V4cGVj
dGVkL2NoZWNrX2J0cmVlLm91dApAQCAtMSwzICsxLDUgQEAKKy0tIGRpcmVjdG9yeSBwYXRo
cyBhcmUgcGFzc2VkIHRvIHVzIGluIGVudmlyb25tZW50IHZhcmlhYmxlcworXGdldGVudiBh
YnNfc3JjZGlyIFBHX0FCU19TUkNESVIKIENSRUFURSBUQUJMRSBidHRlc3RfYShpZCBpbnQ4
KTsKIENSRUFURSBUQUJMRSBidHRlc3RfYihpZCBpbnQ4KTsKIENSRUFURSBUQUJMRSBidHRl
c3RfbXVsdGkoaWQgaW50OCwgZGF0YSBpbnQ4KTsKQEAgLTE5OSw2ICsyMDEsMjIgQEAgU0VM
RUNUIGJ0X2luZGV4X2NoZWNrKCdidHRlc3RfYV9leHByX2lkeCcsIHRydWUpOwogIAogKDEg
cm93KQogCistLQorLS0gQlVHOiBtdXN0IHN1cHBvcnQgZGlmZmVyZW50IGhlYWRlciBzaXpl
IG9mIHNob3J0IHZhcmxlbmEgZGF0dW0KKy0tCitDUkVBVEUgVEFCTEUgdmFybGVuYV9idWcg
KHYgdGV4dCk7CitBTFRFUiBUQUJMRSB2YXJsZW5hX2J1ZyBBTFRFUiBjb2x1bW4gdiBTRVQg
c3RvcmFnZSBwbGFpbjsKK0lOU0VSVCBJTlRPIHZhcmxlbmFfYnVnIFZBTFVFUyAoJ3gnKTsK
K1xzZXQgZmlsZW5hbWUgOmFic19zcmNkaXIgJy9yZXN1bHRzL3ZhcmxlbmFfYnVnLmRtcCcK
K0NPUFkgdmFybGVuYV9idWcgVE8gOidmaWxlbmFtZSc7CitDT1BZIHZhcmxlbmFfYnVnIEZS
T00gOidmaWxlbmFtZSc7CitDUkVBVEUgSU5ERVggdmFybGVuYV9idWdfaWR4IG9uIHZhcmxl
bmFfYnVnKHYpOworU0VMRUNUIGJ0X2luZGV4X2NoZWNrKCd2YXJsZW5hX2J1Z19pZHgnLCB0
cnVlKTsKKyBidF9pbmRleF9jaGVjayAKKy0tLS0tLS0tLS0tLS0tLS0KKyAKKygxIHJvdykK
KwogLS0gY2xlYW51cAogRFJPUCBUQUJMRSBidHRlc3RfYTsKIERST1AgVEFCTEUgYnR0ZXN0
X2I7CkBAIC0yMDgsMyArMjI2LDQgQEAgRFJPUCBUQUJMRSB0b2FzdF9idWc7CiBEUk9QIEZV
TkNUSU9OIGlmdW4oaW50OCk7CiBEUk9QIE9XTkVEIEJZIHJlZ3Jlc3NfYnR0ZXN0X3JvbGU7
IC0tIHBlcm1pc3Npb25zCiBEUk9QIFJPTEUgcmVncmVzc19idHRlc3Rfcm9sZTsKK0RST1Ag
VEFCTEUgdmFybGVuYV9idWc7CmRpZmYgLS1naXQgYS9jb250cmliL2FtY2hlY2svc3FsL2No
ZWNrX2J0cmVlLnNxbCBiL2NvbnRyaWIvYW1jaGVjay9zcWwvY2hlY2tfYnRyZWUuc3FsCmlu
ZGV4IDAzM2MwNGI0ZDA1Li5lZmY2NzlhNzFlZiAxMDA2NDQKLS0tIGEvY29udHJpYi9hbWNo
ZWNrL3NxbC9jaGVja19idHJlZS5zcWwKKysrIGIvY29udHJpYi9hbWNoZWNrL3NxbC9jaGVj
a19idHJlZS5zcWwKQEAgLTEsMyArMSw2IEBACistLSBkaXJlY3RvcnkgcGF0aHMgYXJlIHBh
c3NlZCB0byB1cyBpbiBlbnZpcm9ubWVudCB2YXJpYWJsZXMKK1xnZXRlbnYgYWJzX3NyY2Rp
ciBQR19BQlNfU1JDRElSCisKIENSRUFURSBUQUJMRSBidHRlc3RfYShpZCBpbnQ4KTsKIENS
RUFURSBUQUJMRSBidHRlc3RfYihpZCBpbnQ4KTsKIENSRUFURSBUQUJMRSBidHRlc3RfbXVs
dGkoaWQgaW50OCwgZGF0YSBpbnQ4KTsKQEAgLTEzNSw2ICsxMzgsMTkgQEAgQ1JFQVRFIElO
REVYIGJ0dGVzdF9hX2V4cHJfaWR4IE9OIGJ0dGVzdF9hICgoaWZ1bihpZCkgKyBpZnVuKDAp
KSkKIAogU0VMRUNUIGJ0X2luZGV4X2NoZWNrKCdidHRlc3RfYV9leHByX2lkeCcsIHRydWUp
OwogCistLQorLS0gQlVHOiBtdXN0IHN1cHBvcnQgZGlmZmVyZW50IGhlYWRlciBzaXplIG9m
IHNob3J0IHZhcmxlbmEgZGF0dW0KKy0tCisKK0NSRUFURSBUQUJMRSB2YXJsZW5hX2J1ZyAo
diB0ZXh0KTsKK0FMVEVSIFRBQkxFIHZhcmxlbmFfYnVnIEFMVEVSIGNvbHVtbiB2IFNFVCBz
dG9yYWdlIHBsYWluOworSU5TRVJUIElOVE8gdmFybGVuYV9idWcgVkFMVUVTICgneCcpOwor
XHNldCBmaWxlbmFtZSA6YWJzX3NyY2RpciAnL3Jlc3VsdHMvdmFybGVuYV9idWcuZG1wJwor
Q09QWSB2YXJsZW5hX2J1ZyBUTyA6J2ZpbGVuYW1lJzsKK0NPUFkgdmFybGVuYV9idWcgRlJP
TSA6J2ZpbGVuYW1lJzsKK0NSRUFURSBJTkRFWCB2YXJsZW5hX2J1Z19pZHggb24gdmFybGVu
YV9idWcodik7CitTRUxFQ1QgYnRfaW5kZXhfY2hlY2soJ3ZhcmxlbmFfYnVnX2lkeCcsIHRy
dWUpOworCiAtLSBjbGVhbnVwCiBEUk9QIFRBQkxFIGJ0dGVzdF9hOwogRFJPUCBUQUJMRSBi
dHRlc3RfYjsKQEAgLTE0NCwzICsxNjAsNCBAQCBEUk9QIFRBQkxFIHRvYXN0X2J1ZzsKIERS
T1AgRlVOQ1RJT04gaWZ1bihpbnQ4KTsKIERST1AgT1dORUQgQlkgcmVncmVzc19idHRlc3Rf
cm9sZTsgLS0gcGVybWlzc2lvbnMKIERST1AgUk9MRSByZWdyZXNzX2J0dGVzdF9yb2xlOwor
RFJPUCBUQUJMRSB2YXJsZW5hX2J1ZzsKZGlmZiAtLWdpdCBhL2NvbnRyaWIvYW1jaGVjay92
ZXJpZnlfbmJ0cmVlLmMgYi9jb250cmliL2FtY2hlY2svdmVyaWZ5X25idHJlZS5jCmluZGV4
IDkwMjFkMTU2ZWI3Li4yNTQzZjJjY2ZkMSAxMDA2NDQKLS0tIGEvY29udHJpYi9hbWNoZWNr
L3ZlcmlmeV9uYnRyZWUuYworKysgYi9jb250cmliL2FtY2hlY2svdmVyaWZ5X25idHJlZS5j
CkBAIC0yNTU2LDcgKzI1NTYsNyBAQCBidF9ub3JtYWxpemVfdHVwbGUoQnRyZWVDaGVja1N0
YXRlICpzdGF0ZSwgSW5kZXhUdXBsZSBpdHVwKQogCVR1cGxlRGVzYwl0dXBsZURlc2NyaXB0
b3IgPSBSZWxhdGlvbkdldERlc2NyKHN0YXRlLT5yZWwpOwogCURhdHVtCQlub3JtYWxpemVk
W0lOREVYX01BWF9LRVlTXTsKIAlib29sCQlpc251bGxbSU5ERVhfTUFYX0tFWVNdOwotCWJv
b2wJCXRvYXN0X2ZyZWVbSU5ERVhfTUFYX0tFWVNdOworCWJvb2wJCW5lZWRfZnJlZVtJTkRF
WF9NQVhfS0VZU107CiAJYm9vbAkJZm9ybW5ld3R1cCA9IGZhbHNlOwogCUluZGV4VHVwbGUJ
cmVmb3JtZWQ7CiAJaW50CQkJaTsKQEAgLTI1NzUsNyArMjU3NSw3IEBAIGJ0X25vcm1hbGl6
ZV90dXBsZShCdHJlZUNoZWNrU3RhdGUgKnN0YXRlLCBJbmRleFR1cGxlIGl0dXApCiAJCWF0
dCA9IFR1cGxlRGVzY0F0dHIodHVwbGVEZXNjcmlwdG9yLCBpKTsKIAogCQkvKiBBc3N1bWUg
dW50b2FzdGVkL2FscmVhZHkgbm9ybWFsaXplZCBkYXR1bSBpbml0aWFsbHkgKi8KLQkJdG9h
c3RfZnJlZVtpXSA9IGZhbHNlOworCQluZWVkX2ZyZWVbaV0gPSBmYWxzZTsKIAkJbm9ybWFs
aXplZFtpXSA9IGluZGV4X2dldGF0dHIoaXR1cCwgYXR0LT5hdHRudW0sCiAJCQkJCQkJCQkg
IHR1cGxlRGVzY3JpcHRvciwKIAkJCQkJCQkJCSAgJmlzbnVsbFtpXSk7CkBAIC0yNTg3LDYg
KzI1ODcsNyBAQCBidF9ub3JtYWxpemVfdHVwbGUoQnRyZWVDaGVja1N0YXRlICpzdGF0ZSwg
SW5kZXhUdXBsZSBpdHVwKQogCQkgKiBpbmRleCB3aXRob3V0IGZ1cnRoZXIgcHJvY2Vzc2lu
Zywgc28gYW4gZXh0ZXJuYWwgdmFybGVuYSBoZWFkZXIKIAkJICogc2hvdWxkIG5ldmVyIGJl
IGVuY291bnRlcmVkIGhlcmUKIAkJICovCisKIAkJaWYgKFZBUkFUVF9JU19FWFRFUk5BTChE
YXR1bUdldFBvaW50ZXIobm9ybWFsaXplZFtpXSkpKQogCQkJZXJlcG9ydChFUlJPUiwKIAkJ
CQkJKGVycmNvZGUoRVJSQ09ERV9JTkRFWF9DT1JSVVBURUQpLApAQCAtMjU5OCwxMSArMjU5
OSwzMiBAQCBidF9ub3JtYWxpemVfdHVwbGUoQnRyZWVDaGVja1N0YXRlICpzdGF0ZSwgSW5k
ZXhUdXBsZSBpdHVwKQogCQl7CiAJCQlmb3JtbmV3dHVwID0gdHJ1ZTsKIAkJCW5vcm1hbGl6
ZWRbaV0gPSBQb2ludGVyR2V0RGF0dW0oUEdfREVUT0FTVF9EQVRVTShub3JtYWxpemVkW2ld
KSk7Ci0JCQl0b2FzdF9mcmVlW2ldID0gdHJ1ZTsKKwkJCW5lZWRfZnJlZVtpXSA9IHRydWU7
CiAJCX0KKwkJLyoKKwkJICogU2hvcnQgdHVwbGVzIG1heSBoYXZlIDFCIG9yIDRCIGhlYWRl
ci4gQ29udmVydCA0QiBoZWFkZXIgb2Ygc2hvcnQKKwkJICogdHVwbGVzIHRvIDFCCisJCSAq
LworCQllbHNlIGlmIChWQVJBVFRfQ0FOX01BS0VfU0hPUlQoRGF0dW1HZXRQb2ludGVyKG5v
cm1hbGl6ZWRbaV0pKSkKKwkJeworCQkJLyogY29udmVydCB0byBzaG9ydCB2YXJsZW5hICov
CisJCQlTaXplIGxlbiA9IFZBUkFUVF9DT05WRVJURURfU0hPUlRfU0laRShEYXR1bUdldFBv
aW50ZXIobm9ybWFsaXplZFtpXSkpOworCQkJY2hhciAqZGF0YSA9IHBhbGxvYyhsZW4pOwor
CisJCQlTRVRfVkFSU0laRV9TSE9SVChkYXRhLCBsZW4pOworCQkJbWVtY3B5KGRhdGEgKyAx
LCBWQVJEQVRBKERhdHVtR2V0UG9pbnRlcihub3JtYWxpemVkW2ldKSksIGxlbiAtIDEpOwor
CisJCQlmb3JtbmV3dHVwID0gdHJ1ZTsKKwkJCW5vcm1hbGl6ZWRbaV0gPSBQb2ludGVyR2V0
RGF0dW0oZGF0YSk7CisJCQluZWVkX2ZyZWVbaV0gPSB0cnVlOworCQl9CisKIAl9CiAKLQkv
KiBFYXNpZXIgY2FzZTogVHVwbGUgaGFzIHZhcmxlbmEgZGF0dW1zLCBub25lIG9mIHdoaWNo
IGFyZSBjb21wcmVzc2VkICovCisJLyoKKwkgKiBFYXNpZXIgY2FzZTogVHVwbGUgaGFzIHZh
cmxlbmEgZGF0dW1zLCBub25lIG9mIHdoaWNoIGFyZSBjb21wcmVzc2VkIG9yCisJICogc2hv
cnQgd2l0aCA0QiBoZWFkZXIKKwkgKi8KIAlpZiAoIWZvcm1uZXd0dXApCiAJCXJldHVybiBp
dHVwOwogCkBAIC0yNjExLDYgKzI2MzMsMTAgQEAgYnRfbm9ybWFsaXplX3R1cGxlKEJ0cmVl
Q2hlY2tTdGF0ZSAqc3RhdGUsIEluZGV4VHVwbGUgaXR1cCkKIAkgKiBjcmVhdGluZyBub3Jt
YWxpemVkIHZlcnNpb24gb2YgdGhlIHR1cGxlIGZyb20gdW5jb21wcmVzc2VkIGlucHV0IGRh
dHVtcwogCSAqIChub3JtYWxpemVkIGlucHV0IGRhdHVtcykuICBUaGlzIGlzIHJhdGhlciBu
YWl2ZSwgYnV0IHNob3VsZG4ndCBiZQogCSAqIG5lY2Vzc2FyeSB0b28gb2Z0ZW4uCisJICog
QWxzbyB0dXBsZSBoYWQgc2hvcnQgdmFybGVuYSBkYXR1bXMgd2l0aCA0QiBoZWFkZXIuIEFj
dHVhbGx5IHRoZXJlIGlzIG5vCisJICogcmVzdHJpY3Rpb24gd2l0aCBoYXZlIGhlYXAgdHVw
bGUgY29udGFpbmluZyB2YXJsZW5hIGRhdHVtIHdpdGggNEIgaGVhZGVyCisJICogYW5kIGNv
cnJlc3BvbmRpbmcgaW5kZXggdHVwbGUgY29udGFpbmluZyB2YXJsZW5hIGRhdHVtIHdpdGgg
MUIgaGVhZGVyLgorCSAqIEZvciBmaW5nZXJwcmludGluZyBsZXQncyBjb252ZXJ0IGhlYXAg
dHVwbGUgdmFybGVuYSBkYXR1bSB0byAxQiBmb3JtYXQuCiAJICoKIAkgKiBOb3RlIHRoYXQg
d2UgcmVseSBvbiBkZXRlcm1pbmlzdGljIGluZGV4X2Zvcm1fdHVwbGUoKSBUT0FTVCBjb21w
cmVzc2lvbgogCSAqIG9mIG5vcm1hbGl6ZWQgaW5wdXQuCkBAIC0yNjIwLDcgKzI2NDYsNyBA
QCBidF9ub3JtYWxpemVfdHVwbGUoQnRyZWVDaGVja1N0YXRlICpzdGF0ZSwgSW5kZXhUdXBs
ZSBpdHVwKQogCiAJLyogQ2Fubm90IGxlYWsgbWVtb3J5IGhlcmUgKi8KIAlmb3IgKGkgPSAw
OyBpIDwgdHVwbGVEZXNjcmlwdG9yLT5uYXR0czsgaSsrKQotCQlpZiAodG9hc3RfZnJlZVtp
XSkKKwkJaWYgKG5lZWRfZnJlZVtpXSkKIAkJCXBmcmVlKERhdHVtR2V0UG9pbnRlcihub3Jt
YWxpemVkW2ldKSk7CiAKIAlyZXR1cm4gcmVmb3JtZWQ7Ci0tIAoyLjQzLjAKCg==

--------------s8TMrgCqChb4s73L1OMfgQVt--