From: Tom Lane <tgl@sss.pgh.pa.us>
To: Justin Pryzby <pryzby@telsasoft.com>
cc: Dilip Kumar <dilipbalaut@gmail.com>, Robert Haas <robertmhaas@gmail.com>,
        Andres Freund <andres@anarazel.de>,
        Tomas Vondra <tomas.vondra@2ndquadrant.com>,
        Alexander Korotkov <a.korotkov@postgrespro.ru>,
        David Steele <david@pgmasters.net>,
        Ildus Kurbangaliev <i.kurbangaliev@gmail.com>,
        Dmitry Dolgov <9erthalion6@gmail.com>,
        Andrey Borodin <x4mmm@yandex-team.ru>,
        PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: [HACKERS] Custom compression methods (mac+lz4.h)
In-reply-to: <20210321234324.GC4203@telsasoft.com>
References: <301730.1616276227@sss.pgh.pa.us>
 <20210320231957.GX11765@telsasoft.com> <309337.1616287493@sss.pgh.pa.us>
 <311370.1616290403@sss.pgh.pa.us>
 <CAFiTN-tN-fi=9VEatyiZyhW28yhhQCmn-egYzbdxBTLkOnVQvg@mail.gmail.com>
 <CAFiTN-t0w+Rc2U3S+y=7KWcLuOYNB5MfWeGdNa7+pg0UovVdcQ@mail.gmail.com>
 <414943.1616358751@sss.pgh.pa.us> <20210321204148.GB4203@telsasoft.com>
 <456616.1616360715@sss.pgh.pa.us> <504396.1616368310@sss.pgh.pa.us>
 <20210321234324.GC4203@telsasoft.com>
Comments: In-reply-to Justin Pryzby <pryzby@telsasoft.com>
	message dated "Sun, 21 Mar 2021 18:43:24 -0500"
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0"
Content-ID: <507544.1616370649.0@sss.pgh.pa.us>
Date: Sun, 21 Mar 2021 19:52:09 -0400
Message-ID: <507597.1616370729@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/507597.1616370729%40sss.pgh.pa.us>
Precedence: bulk

------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <507544.1616370649.1@sss.pgh.pa.us>

Justin Pryzby <pryzby@telsasoft.com> writes:
> On Sun, Mar 21, 2021 at 07:11:50PM -0400, Tom Lane wrote:
>> I hate to be the bearer of bad news, but this suggests that
>> LZ4_decompress_safe_partial is seriously broken in 1.9.2
>> as well:
>> https://github.com/lz4/lz4/issues/783

> Ouch

Actually, after reading that closer, the problem only affects the
case where the compressed-data-length passed to the function is
a lie.  So it shouldn't be a problem for our usage.

Also, after studying the documentation for LZ4_decompress_safe
and LZ4_decompress_safe_partial, I realized that liblz4 is also
counting on the *output* buffer size to not be a lie.  So we
cannot pass it a number larger than the chunk's true decompressed
size.  The attached patch resolves the issue I'm seeing.

			regards, tom lane

------- =_aaaaaaaaaa0
Content-Type: text/x-diff; name="fix-partial-lz4-decompression.patch";
	charset="us-ascii"
Content-ID: <507544.1616370649.2@sss.pgh.pa.us>
Content-Description: fix-partial-lz4-decompression.patch
Content-Transfer-Encoding: quoted-printable

diff --git a/src/backend/access/common/toast_compression.c b/src/backend/a=
ccess/common/toast_compression.c
index 00af1740cf..74e449992a 100644
--- a/src/backend/access/common/toast_compression.c
+++ b/src/backend/access/common/toast_compression.c
@@ -220,6 +220,10 @@ lz4_decompress_datum_slice(const struct varlena *valu=
e, int32 slicelength)
 	if (LZ4_versionNumber() < 10803)
 		return lz4_decompress_datum(value);
 =

+	/* liblz4 assumes that slicelength is not an overestimate */
+	if (slicelength >=3D VARRAWSIZE_4B_C(value))
+		return lz4_decompress_datum(value);
+
 	/* allocate memory for the uncompressed data */
 	result =3D (struct varlena *) palloc(slicelength + VARHDRSZ);
 =


------- =_aaaaaaaaaa0--