MIME-Version: 1.0
From: SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com>
Date: Wed, 15 Apr 2026 23:13:42 -0700
Message-ID: 
 <CAHg+QDeXb9HM2VGKXQedyCp52GzajJK5KOUdNi6oLjsS0nerQw@mail.gmail.com>
Subject: [PATCH] Compressed TOAST data corruption with REPACK CONCURRENTLY
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
	=?UTF-8?Q?=C3=81lvaro_Herrera?= <alvherre@kurilemu.de>, ah@cybertec.at
Content-Type: multipart/mixed; boundary="000000000000a34894064f8dbc67"
Archived-At: 
 <https://www.postgresql.org/message-id/CAHg%2BQDeXb9HM2VGKXQedyCp52GzajJK5KOUdNi6oLjsS0nerQw%40mail.gmail.com>
Precedence: bulk

--000000000000a34894064f8dbc67
Content-Type: multipart/alternative; boundary="000000000000a34891064f8dbc65"

--000000000000a34891064f8dbc65
Content-Type: text/plain; charset="UTF-8"

Hi hackers,

restore_tuple() in repack.c uses SET_VARSIZE() to reconstruct the varlena
header when
reading back external attributes from the spill file. In this process,
looks like the flag
SET_VARSIZE_COMPRESSED is silently lost. Because of this, when REPACK
CONCURRENTLY
run  any concurrently updated column whose value was TOAST-compressed ends
up with raw
compressed bytes behind an "uncompressed" header returning garbled data on
subsequent reads.
It appears that existing tests are using random chars which are
uncompressable.

Please find the
attached 0001-Fix-restore_tuple-losing-varlena-compression-flag.patch to
fix this.
Additionally I updated the existing repack_toast test to include the
scenario I was talking about.

Thanks,
Satya

--000000000000a34891064f8dbc65
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi hackers,<div><br></div><div>restore_tuple() in repack.c=
 uses SET_VARSIZE() to reconstruct the varlena header when=C2=A0</div><div>=
reading back external attributes from the spill file. In this process, look=
s like the flag=C2=A0</div><div>SET_VARSIZE_COMPRESSED is silently lost. Be=
cause of this, when REPACK CONCURRENTLY</div><div>run=C2=A0 any concurrentl=
y updated=C2=A0column whose value was TOAST-compressed ends up with raw=C2=
=A0</div><div>compressed bytes behind an &quot;uncompressed&quot; header re=
turning garbled data on subsequent reads.</div><div>It appears that existin=
g tests are using random chars which are uncompressable.</div><div><br></di=
v><div>Please find the attached=C2=A00001-Fix-restore_tuple-losing-varlena-=
compression-flag.patch to fix this.=C2=A0</div><div>Additionally I updated =
the existing repack_toast test to include the scenario I was talking about.=
</div><div><br></div><div>Thanks,</div><div>Satya</div></div>

--000000000000a34891064f8dbc65--

--000000000000a34894064f8dbc67
Content-Type: application/octet-stream;
	name="0001-Fix-restore_tuple-losing-varlena-compression-flag.patch"
Content-Disposition: attachment;
	filename="0001-Fix-restore_tuple-losing-varlena-compression-flag.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mo12m3s60>
X-Attachment-Id: f_mo12m3s60

ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2NvbW1hbmRzL3JlcGFjay5jIGIvc3JjL2JhY2tlbmQv
Y29tbWFuZHMvcmVwYWNrLmMKaW5kZXggNThlMzg2Ny4uZTZiNmZlYSAxMDA2NDQKLS0tIGEvc3Jj
L2JhY2tlbmQvY29tbWFuZHMvcmVwYWNrLmMKKysrIGIvc3JjL2JhY2tlbmQvY29tbWFuZHMvcmVw
YWNrLmMKQEAgLTI3MjcsNyArMjcyNywxMCBAQCByZXN0b3JlX3R1cGxlKEJ1ZkZpbGUgKmZpbGUs
IFJlbGF0aW9uIHJlbGF0aW9uLCBUdXBsZVRhYmxlU2xvdCAqc2xvdCkKIAkJCXZhcmxlbnN6ID0g
VkFSU0laRV9BTlkoJmNodW5rX2hlYWRlcik7CiAKIAkJCXZhbHVlID0gcGFsbG9jKHZhcmxlbnN6
KTsKLQkJCVNFVF9WQVJTSVpFKHZhbHVlLCBWQVJTSVpFX0FOWSgmY2h1bmtfaGVhZGVyKSk7CisJ
CQlpZiAoVkFSQVRUX0lTXzRCX0MoJmNodW5rX2hlYWRlcikpCisJCQkJU0VUX1ZBUlNJWkVfQ09N
UFJFU1NFRCh2YWx1ZSwgdmFybGVuc3opOworCQkJZWxzZQorCQkJCVNFVF9WQVJTSVpFKHZhbHVl
LCB2YXJsZW5zeik7CiAJCQlCdWZGaWxlUmVhZEV4YWN0KGZpbGUsIChjaGFyICopIHZhbHVlICsg
VkFSSERSU1osIHZhcmxlbnN6IC0gVkFSSERSU1opOwogCiAJCQlzbG90LT50dHNfdmFsdWVzW2ld
ID0gUG9pbnRlckdldERhdHVtKHZhbHVlKTsK
--000000000000a34894064f8dbc67
Content-Type: application/octet-stream;
	name="0002-Add-compressed-TOAST-test-to-repack_toast.patch"
Content-Disposition: attachment;
	filename="0002-Add-compressed-TOAST-test-to-repack_toast.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mo1312s61>
X-Attachment-Id: f_mo1312s61

ZGlmZiAtLWdpdCBhL3NyYy90ZXN0L21vZHVsZXMvaW5qZWN0aW9uX3BvaW50cy9leHBlY3RlZC9y
ZXBhY2tfdG9hc3Qub3V0IGIvc3JjL3Rlc3QvbW9kdWxlcy9pbmplY3Rpb25fcG9pbnRzL2V4cGVj
dGVkL3JlcGFja190b2FzdC5vdXQKaW5kZXggYjU2ZGRlMS4uMGVlZmM4OCAxMDA2NDQKLS0tIGEv
c3JjL3Rlc3QvbW9kdWxlcy9pbmplY3Rpb25fcG9pbnRzL2V4cGVjdGVkL3JlcGFja190b2FzdC5v
dXQKKysrIGIvc3JjL3Rlc3QvbW9kdWxlcy9pbmplY3Rpb25fcG9pbnRzL2V4cGVjdGVkL3JlcGFj
a190b2FzdC5vdXQKQEAgLTYzLDMgKzYzLDYwIEBAIGluamVjdGlvbl9wb2ludHNfZGV0YWNoCiAg
ICAgICAgICAgICAgICAgICAgICAgIAogKDEgcm93KQogCisKK3N0YXJ0aW5nIHBlcm11dGF0aW9u
OiByZXBhY2tfY29tcHJlc3NlZCBjb21wcmVzc2VkX3VwZGF0ZSB3YWtldXBfYmVmb3JlX2xvY2sg
dmVyaWZ5X2NvbXByZXNzZWQKK2luamVjdGlvbl9wb2ludHNfYXR0YWNoCistLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLQorICAgICAgICAgICAgICAgICAgICAgICAKKygxIHJvdykKKworc3RlcCByZXBh
Y2tfY29tcHJlc3NlZDogCisJUkVQQUNLIChDT05DVVJSRU5UTFkpIHJlcGFja19jdGVzdDsKKyA8
d2FpdGluZyAuLi4+CitzdGVwIGNvbXByZXNzZWRfdXBkYXRlOiAKKwlVUERBVEUgcmVwYWNrX2N0
ZXN0IFNFVCBqID0gZ2VuX2NvbXByZXNzaWJsZSg5OSkgV0hFUkUgaSA9IDI7CisKK3N0ZXAgd2Fr
ZXVwX2JlZm9yZV9sb2NrOiAKKwlTRUxFQ1QgaW5qZWN0aW9uX3BvaW50c193YWtldXAoJ3JlcGFj
ay1jb25jdXJyZW50bHktYmVmb3JlLWxvY2snKTsKKworaW5qZWN0aW9uX3BvaW50c193YWtldXAK
Ky0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCisgICAgICAgICAgICAgICAgICAgICAgIAorKDEgcm93
KQorCitzdGVwIHJlcGFja19jb21wcmVzc2VkOiA8Li4uIGNvbXBsZXRlZD4KK3N0ZXAgdmVyaWZ5
X2NvbXByZXNzZWQ6IAorCUlOU0VSVCBJTlRPIHBvc3RfcmVwYWNrX2RhdGEKKwlTRUxFQ1QgaSwg
aiBGUk9NIHJlcGFja19jdGVzdCBPUkRFUiBCWSBpOworCisJU0VMRUNUIGUuaSwKKwkgICAgICAg
bGVuZ3RoKGUuaikgQVMgZXhwZWN0X2xlbiwKKwkgICAgICAgbGVuZ3RoKHAuaikgQVMgYWN0dWFs
X2xlbiwKKwkgICAgICAgKGxlbmd0aChlLmopID0gbGVuZ3RoKHAuaikpIEFTIGxlbl9vaywKKwkg
ICAgICAgKG1kNShlLmopID0gbWQ1KHAuaikpIEFTIGhhc2hfb2sKKwlGUk9NIGV4cGVjdGVkX2Rh
dGEgZQorCUpPSU4gcG9zdF9yZXBhY2tfZGF0YSBwIFVTSU5HIChpKQorCU9SREVSIEJZIGUuaTsK
KworCVNFTEVDVCBjb3VudCgqKSBBUyBtaXNtYXRjaGVzCisJRlJPTSBleHBlY3RlZF9kYXRhIGUK
KwlKT0lOIHBvc3RfcmVwYWNrX2RhdGEgcCBVU0lORyAoaSkKKwlXSEVSRSBsZW5ndGgoZS5qKSAh
PSBsZW5ndGgocC5qKSBPUiBtZDUoZS5qKSAhPSBtZDUocC5qKTsKKworaXxleHBlY3RfbGVufGFj
dHVhbF9sZW58bGVuX29rfGhhc2hfb2sKKy0rLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLSst
LS0tLS0tCisxfCAgICAgMTYwMDB8ICAgICAxNjAwMHx0ICAgICB8dCAgICAgIAorMnwgICAgIDE2
MDAwfCAgICAgMTYwMDB8dCAgICAgfHQgICAgICAKKzN8ICAgICAxNjAwMHwgICAgIDE2MDAwfHQg
ICAgIHx0ICAgICAgCisoMyByb3dzKQorCittaXNtYXRjaGVzCistLS0tLS0tLS0tCisgICAgICAg
ICAwCisoMSByb3cpCisKK2luamVjdGlvbl9wb2ludHNfZGV0YWNoCistLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLQorICAgICAgICAgICAgICAgICAgICAgICAKKygxIHJvdykKKwpkaWZmIC0tZ2l0IGEv
c3JjL3Rlc3QvbW9kdWxlcy9pbmplY3Rpb25fcG9pbnRzL3NwZWNzL3JlcGFja190b2FzdC5zcGVj
IGIvc3JjL3Rlc3QvbW9kdWxlcy9pbmplY3Rpb25fcG9pbnRzL3NwZWNzL3JlcGFja190b2FzdC5z
cGVjCmluZGV4IGI4NzhiMTkuLmZiOGE1NTcgMTAwNjQ0Ci0tLSBhL3NyYy90ZXN0L21vZHVsZXMv
aW5qZWN0aW9uX3BvaW50cy9zcGVjcy9yZXBhY2tfdG9hc3Quc3BlYworKysgYi9zcmMvdGVzdC9t
b2R1bGVzL2luamVjdGlvbl9wb2ludHMvc3BlY3MvcmVwYWNrX3RvYXN0LnNwZWMKQEAgLTEzLDYg
KzEzLDE0IEBAIHNldHVwCiAJCUZST00gZ2VuZXJhdGVfc2VyaWVzKDEsIDIwNDgpIHMoeCk7CiAJ
JCQ7CiAKKwktLSA1MDAgY29uY2F0ZW5hdGVkIG1kNSBoYXNoZXMgPSAxNjAwMCBoZXggY2hhcnM7
IGNvbXByZXNzZXMgYnV0IHN0YXlzID4gMktCLgorCUNSRUFURSBGVU5DVElPTiBnZW5fY29tcHJl
c3NpYmxlKHNlZWQgaW50KQorCVJFVFVSTlMgdGV4dAorCUxBTkdVQUdFIHNxbCBJTU1VVEFCTEUg
YXMgJCQKKwkJU0VMRUNUIHN0cmluZ19hZ2cobWQ1KChzZWVkICogMTAwMCArIHgpOjp0ZXh0KSwg
JycpCisJCUZST00gZ2VuZXJhdGVfc2VyaWVzKDEsIDUwMCkgeDsKKwkkJDsKKwogCUNSRUFURSBU
QUJMRSByZXBhY2tfdGVzdChpIGludCBQUklNQVJZIEtFWSwgaiB0ZXh0KTsKIAlJTlNFUlQgSU5U
TyByZXBhY2tfdGVzdChpLCBqKSBWQUxVRVMgKDEsIGdldF9sb25nX3N0cmluZygpKSwKIAkJKDIs
IGdldF9sb25nX3N0cmluZygpKSwgKDMsIGdldF9sb25nX3N0cmluZygpKTsKQEAgLTIxLDEzICsy
OSwzMCBAQCBzZXR1cAogCiAJQ1JFQVRFIFRBQkxFIGRhdGFfczEoaSBpbnQsIGogdGV4dCk7CiAJ
Q1JFQVRFIFRBQkxFIGRhdGFfczIoaSBpbnQsIGogdGV4dCk7CisKKwlDUkVBVEUgVEFCTEUgcmVw
YWNrX2N0ZXN0KGkgaW50IFBSSU1BUlkgS0VZLCBqIHRleHQpOworCUlOU0VSVCBJTlRPIHJlcGFj
a19jdGVzdChpLCBqKQorCVZBTFVFUyAoMSwgZ2VuX2NvbXByZXNzaWJsZSgxKSksCisJICAgICAg
ICgyLCBnZW5fY29tcHJlc3NpYmxlKDIpKSwKKwkgICAgICAgKDMsIGdlbl9jb21wcmVzc2libGUo
MykpOworCisJQ1JFQVRFIFRBQkxFIGV4cGVjdGVkX2RhdGEgQVMKKwlTRUxFQ1QgMSBBUyBpLCBn
ZW5fY29tcHJlc3NpYmxlKDEpIEFTIGoKKwlVTklPTiBBTEwgU0VMRUNUIDIsIGdlbl9jb21wcmVz
c2libGUoOTkpCisJVU5JT04gQUxMIFNFTEVDVCAzLCBnZW5fY29tcHJlc3NpYmxlKDMpOworCisJ
Q1JFQVRFIFRBQkxFIHBvc3RfcmVwYWNrX2RhdGEoaSBpbnQsIGogdGV4dCk7CiB9CiAKIHRlYXJk
b3duCiB7CiAJRFJPUCBUQUJMRSByZXBhY2tfdGVzdDsKKwlEUk9QIFRBQkxFIHJlcGFja19jdGVz
dDsKKwlEUk9QIFRBQkxFIGV4cGVjdGVkX2RhdGE7CisJRFJPUCBUQUJMRSBwb3N0X3JlcGFja19k
YXRhOwogCURST1AgRVhURU5TSU9OIGluamVjdGlvbl9wb2ludHM7CiAJRFJPUCBGVU5DVElPTiBn
ZXRfbG9uZ19zdHJpbmcoKTsKKwlEUk9QIEZVTkNUSU9OIGdlbl9jb21wcmVzc2libGUoaW50KTsK
IAogCURST1AgVEFCTEUgcmVsZmlsZW5vZGVzOwogCURST1AgVEFCTEUgZGF0YV9zMTsKQEAgLTY3
LDYgKzkyLDI5IEBAIHN0ZXAgY2hlY2sxCiAJRlJPTSBkYXRhX3MxIGQxIEZVTEwgSk9JTiBkYXRh
X3MyIGQyIFVTSU5HIChpLCBqKQogCVdIRVJFIGQxLmkgSVNOVUxMIE9SIGQyLmkgSVNOVUxMOwog
fQorc3RlcCByZXBhY2tfY29tcHJlc3NlZAoreworCVJFUEFDSyAoQ09OQ1VSUkVOVExZKSByZXBh
Y2tfY3Rlc3Q7Cit9CitzdGVwIHZlcmlmeV9jb21wcmVzc2VkCit7CisJSU5TRVJUIElOVE8gcG9z
dF9yZXBhY2tfZGF0YQorCVNFTEVDVCBpLCBqIEZST00gcmVwYWNrX2N0ZXN0IE9SREVSIEJZIGk7
CisKKwlTRUxFQ1QgZS5pLAorCSAgICAgICBsZW5ndGgoZS5qKSBBUyBleHBlY3RfbGVuLAorCSAg
ICAgICBsZW5ndGgocC5qKSBBUyBhY3R1YWxfbGVuLAorCSAgICAgICAobGVuZ3RoKGUuaikgPSBs
ZW5ndGgocC5qKSkgQVMgbGVuX29rLAorCSAgICAgICAobWQ1KGUuaikgPSBtZDUocC5qKSkgQVMg
aGFzaF9vaworCUZST00gZXhwZWN0ZWRfZGF0YSBlCisJSk9JTiBwb3N0X3JlcGFja19kYXRhIHAg
VVNJTkcgKGkpCisJT1JERVIgQlkgZS5pOworCisJU0VMRUNUIGNvdW50KCopIEFTIG1pc21hdGNo
ZXMKKwlGUk9NIGV4cGVjdGVkX2RhdGEgZQorCUpPSU4gcG9zdF9yZXBhY2tfZGF0YSBwIFVTSU5H
IChpKQorCVdIRVJFIGxlbmd0aChlLmopICE9IGxlbmd0aChwLmopIE9SIG1kNShlLmopICE9IG1k
NShwLmopOworfQogdGVhcmRvd24KIHsKICAgICBTRUxFQ1QgaW5qZWN0aW9uX3BvaW50c19kZXRh
Y2goJ3JlcGFjay1jb25jdXJyZW50bHktYmVmb3JlLWxvY2snKTsKQEAgLTEwMSw2ICsxNDksMTAg
QEAgc3RlcCB3YWtldXBfYmVmb3JlX2xvY2sKIHsKIAlTRUxFQ1QgaW5qZWN0aW9uX3BvaW50c193
YWtldXAoJ3JlcGFjay1jb25jdXJyZW50bHktYmVmb3JlLWxvY2snKTsKIH0KK3N0ZXAgY29tcHJl
c3NlZF91cGRhdGUKK3sKKwlVUERBVEUgcmVwYWNrX2N0ZXN0IFNFVCBqID0gZ2VuX2NvbXByZXNz
aWJsZSg5OSkgV0hFUkUgaSA9IDI7Cit9CiAKICMgVGVzdCBpZiBkYXRhIGNoYW5nZXMgaW50cm9k
dWNlZCB3aGlsZSBvbmUgc2Vzc2lvbiBpcyBwZXJmb3JtaW5nIFJFUEFDSwogIyBDT05DVVJSRU5U
TFkgZmluZCB0aGVpciB3YXkgaW50byB0aGUgdGFibGUuCkBAIC0xMTAsMyArMTYyLDEwIEBAIHBl
cm11dGF0aW9uCiAJY2hlY2syCiAJd2FrZXVwX2JlZm9yZV9sb2NrCiAJY2hlY2sxCisKKyMgVGVz
dCB0aGF0IGNvbXByZXNzZWQgVE9BU1QgdmFsdWVzIHN1cnZpdmUgUkVQQUNLIENPTkNVUlJFTlRM
WS4KK3Blcm11dGF0aW9uCisJcmVwYWNrX2NvbXByZXNzZWQKKwljb21wcmVzc2VkX3VwZGF0ZQor
CXdha2V1cF9iZWZvcmVfbG9jaworCXZlcmlmeV9jb21wcmVzc2VkCg==
--000000000000a34894064f8dbc67--