From: Tom Lane <tgl@sss.pgh.pa.us>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>
Cc: David Steele <david@pgmasters.net>
Cc: Ildus Kurbangaliev <i.kurbangaliev@gmail.com>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: [HACKERS] Custom compression methods (mac+lz4.h)
Date: Mon, 22 Mar 2021 10:41:48 -0400
Message-ID: <558766.1616424108@sss.pgh.pa.us> (raw)
In-Reply-To: <CAFiTN-u1Y2i7TFfC+wZAEiQLHTYMSQafbRisYPqCaU6gQdnPcw@mail.gmail.com>
References: <301730.1616276227@sss.pgh.pa.us>
<20210320231957.GX11765@telsasoft.com>
<309337.1616287493@sss.pgh.pa.us>
<311370.1616290403@sss.pgh.pa.us>
<CAFiTN-tN-fi=9VEatyiZyhW28yhhQCmn-egYzbdxBTLkOnVQvg@mail.gmail.com>
<CAFiTN-t0w+Rc2U3S+y=7KWcLuOYNB5MfWeGdNa7+pg0UovVdcQ@mail.gmail.com>
<414943.1616358751@sss.pgh.pa.us>
<20210321204148.GB4203@telsasoft.com>
<456616.1616360715@sss.pgh.pa.us>
<504396.1616368310@sss.pgh.pa.us>
<20210321234324.GC4203@telsasoft.com>
<507597.1616370729@sss.pgh.pa.us>
<CAFiTN-u1Y2i7TFfC+wZAEiQLHTYMSQafbRisYPqCaU6gQdnPcw@mail.gmail.com>
Dilip Kumar <dilipbalaut@gmail.com> writes:
> On Mon, Mar 22, 2021 at 5:22 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Also, after studying the documentation for LZ4_decompress_safe
>> and LZ4_decompress_safe_partial, I realized that liblz4 is also
>> counting on the *output* buffer size to not be a lie. So we
>> cannot pass it a number larger than the chunk's true decompressed
>> size. The attached patch resolves the issue I'm seeing.
> Okay, the fix makes sense. In fact, IMHO, in general also this fix
> looks like an optimization, I mean when slicelength >=
> VARRAWSIZE_4B_C(value), then why do we need to allocate extra memory
> even in the case of pglz. So shall we put this check directly in
> toast_decompress_datum_slice instead of handling it at the lz4 level?
Yeah, I thought about that too, but do we want to assume that
VARRAWSIZE_4B_C is the correct way to get the decompressed size
for all compression methods?
(If so, I think it would be better style to have a less opaque macro
name for the purpose.)
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us, dilipbalaut@gmail.com, pryzby@telsasoft.com, robertmhaas@gmail.com, andres@anarazel.de, tomas.vondra@2ndquadrant.com, a.korotkov@postgrespro.ru, david@pgmasters.net, i.kurbangaliev@gmail.com, 9erthalion6@gmail.com, x4mmm@yandex-team.ru, pgsql-hackers@lists.postgresql.org
Subject: Re: [HACKERS] Custom compression methods (mac+lz4.h)
In-Reply-To: <558766.1616424108@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox