From: Tom Lane <tgl@sss.pgh.pa.us>
To: Dilip Kumar <dilipbalaut@gmail.com>
cc: Justin Pryzby <pryzby@telsasoft.com>, Robert Haas <robertmhaas@gmail.com>,
        Andres Freund <andres@anarazel.de>,
        Tomas Vondra <tomas.vondra@2ndquadrant.com>,
        Alexander Korotkov <a.korotkov@postgrespro.ru>,
        David Steele <david@pgmasters.net>,
        Ildus Kurbangaliev <i.kurbangaliev@gmail.com>,
        Dmitry Dolgov <9erthalion6@gmail.com>,
        Andrey Borodin <x4mmm@yandex-team.ru>,
        PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: [HACKERS] Custom compression methods (mac+lz4.h)
In-reply-to: 
 <CAFiTN-u1Y2i7TFfC+wZAEiQLHTYMSQafbRisYPqCaU6gQdnPcw@mail.gmail.com>
References: <301730.1616276227@sss.pgh.pa.us>
 <20210320231957.GX11765@telsasoft.com> <309337.1616287493@sss.pgh.pa.us>
 <311370.1616290403@sss.pgh.pa.us>
 <CAFiTN-tN-fi=9VEatyiZyhW28yhhQCmn-egYzbdxBTLkOnVQvg@mail.gmail.com>
 <CAFiTN-t0w+Rc2U3S+y=7KWcLuOYNB5MfWeGdNa7+pg0UovVdcQ@mail.gmail.com>
 <414943.1616358751@sss.pgh.pa.us> <20210321204148.GB4203@telsasoft.com>
 <456616.1616360715@sss.pgh.pa.us> <504396.1616368310@sss.pgh.pa.us>
 <20210321234324.GC4203@telsasoft.com> <507597.1616370729@sss.pgh.pa.us>
 <CAFiTN-u1Y2i7TFfC+wZAEiQLHTYMSQafbRisYPqCaU6gQdnPcw@mail.gmail.com>
Comments: In-reply-to Dilip Kumar <dilipbalaut@gmail.com>
	message dated "Mon, 22 Mar 2021 13:20:47 +0530"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <558765.1616424108.1@sss.pgh.pa.us>
Date: Mon, 22 Mar 2021 10:41:48 -0400
Message-ID: <558766.1616424108@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/558766.1616424108%40sss.pgh.pa.us>
Precedence: bulk

Dilip Kumar <dilipbalaut@gmail.com> writes:
> On Mon, Mar 22, 2021 at 5:22 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Also, after studying the documentation for LZ4_decompress_safe
>> and LZ4_decompress_safe_partial, I realized that liblz4 is also
>> counting on the *output* buffer size to not be a lie.  So we
>> cannot pass it a number larger than the chunk's true decompressed
>> size.  The attached patch resolves the issue I'm seeing.

> Okay, the fix makes sense.  In fact, IMHO, in general also this fix
> looks like an optimization, I mean when slicelength >=
> VARRAWSIZE_4B_C(value), then why do we need to allocate extra memory
> even in the case of pglz.  So shall we put this check directly in
> toast_decompress_datum_slice instead of handling it at the lz4 level?

Yeah, I thought about that too, but do we want to assume that
VARRAWSIZE_4B_C is the correct way to get the decompressed size
for all compression methods?

(If so, I think it would be better style to have a less opaque macro
name for the purpose.)

			regards, tom lane