From: Justin Pryzby <pryzby@telsasoft.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>
Cc: David Steele <david@pgmasters.net>
Cc: Ildus Kurbangaliev <i.kurbangaliev@gmail.com>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: [HACKERS] Custom compression methods
Date: Sun, 28 Feb 2021 23:36:17 -0600
Message-ID: <20210301053617.GA3701@telsasoft.com> (raw)
In-Reply-To: <CAFiTN-t1DEt=K0Wj+mFvkA_y_OFeTNy=G+iZ6pwqhd62iGm0KA@mail.gmail.com>
References: <20210218211314.GC14772@telsasoft.com>
<CAFiTN-u2pyXDDDwZXJ-fVUwbLhJSe9TbrVR6rfW_rhdyL1A5bg@mail.gmail.com>
<CA+TgmoaXs3_uoyOeJhJaS+oY6BvpGmHor70kaS6_03kssM7bVw@mail.gmail.com>
<CAFiTN-t=nNd6-ZyWP=5TQtqvWW0_VhdzQfcMsQYRdEa7eAX0gA@mail.gmail.com>
<CAFiTN-vcVGZrK_T=Oi9vVf=-N=xc0WoK7HiywHf8F9Jap5-hKw@mail.gmail.com>
<CAFiTN-u3jrRDiuyGxGvNSGDXLhG4=U2orHnjk4B6WBy9Eo9kMQ@mail.gmail.com>
<CAFiTN-uaZ8X=sXu6V_EYt8cNE2mWrehKMH3qGoAxzjepPuZGkA@mail.gmail.com>
<CAFiTN-sTcQq2iszeTxZum4x+0jK1x-PHfVDJqUAQCsCTBEeDQQ@mail.gmail.com>
<20210228041821.GB20769@telsasoft.com>
<CAFiTN-t1DEt=K0Wj+mFvkA_y_OFeTNy=G+iZ6pwqhd62iGm0KA@mail.gmail.com>
On Mon, Mar 01, 2021 at 10:32:23AM +0530, Dilip Kumar wrote:
> On Sun, Feb 28, 2021 at 9:48 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> >
> > On my PC, this new change is causing a test failure:
> >
> > SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
> > - substr
> > -----------------------------------------------------
> > - 01234567890123456789012345678901234567890123456789
> > -(1 row)
> > -
> > +ERROR: compressed lz4 data is corrupt
>
> The older version of lz4 had this problem that while decompressing
> partial if we don't give the buffer size up to full data length it was
> failing[1] but it is solved in version 1.9.
Thanks. It seems like that explains it.
I think if that's a problem with recent versions, then you'll have to
conditionally disable slicing.
https://packages.debian.org/liblz4-dev
Slicing isn't generally usable if it sometimes makes people's data inaccessible
and gives errors about corruption.
I guess you could make it a compile time test on these constants (I don't know
the necessary version, though)
#define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */
#define LZ4_VERSION_MINOR 7 /* for new (non-breaking) interface capabilities */
#define LZ4_VERSION_RELEASE 1 /* for tweaks, bug-fixes, or development */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)
If the version is too low, either make it #error, or disable slicing.
The OS usual library version infrastructure will make sure the runtime version
is at least the MAJOR+MINOR of the compile time version.
--
Justin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, dilipbalaut@gmail.com, robertmhaas@gmail.com, tomas.vondra@2ndquadrant.com, a.korotkov@postgrespro.ru, david@pgmasters.net, i.kurbangaliev@gmail.com, 9erthalion6@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: [HACKERS] Custom compression methods
In-Reply-To: <20210301053617.GA3701@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox