Date: Sun, 28 Feb 2021 23:36:17 -0600
From: Justin Pryzby <pryzby@telsasoft.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>,
	Tomas Vondra <tomas.vondra@2ndquadrant.com>,
	Alexander Korotkov <a.korotkov@postgrespro.ru>,
	David Steele <david@pgmasters.net>,
	Ildus Kurbangaliev <i.kurbangaliev@gmail.com>,
	Dmitry Dolgov <9erthalion6@gmail.com>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: [HACKERS] Custom compression methods
Message-ID: <20210301053617.GA3701@telsasoft.com>
References: <20210218211314.GC14772@telsasoft.com>
 <CAFiTN-u2pyXDDDwZXJ-fVUwbLhJSe9TbrVR6rfW_rhdyL1A5bg@mail.gmail.com>
 <CA+TgmoaXs3_uoyOeJhJaS+oY6BvpGmHor70kaS6_03kssM7bVw@mail.gmail.com>
 <CAFiTN-t=nNd6-ZyWP=5TQtqvWW0_VhdzQfcMsQYRdEa7eAX0gA@mail.gmail.com>
 <CAFiTN-vcVGZrK_T=Oi9vVf=-N=xc0WoK7HiywHf8F9Jap5-hKw@mail.gmail.com>
 <CAFiTN-u3jrRDiuyGxGvNSGDXLhG4=U2orHnjk4B6WBy9Eo9kMQ@mail.gmail.com>
 <CAFiTN-uaZ8X=sXu6V_EYt8cNE2mWrehKMH3qGoAxzjepPuZGkA@mail.gmail.com>
 <CAFiTN-sTcQq2iszeTxZum4x+0jK1x-PHfVDJqUAQCsCTBEeDQQ@mail.gmail.com>
 <20210228041821.GB20769@telsasoft.com>
 <CAFiTN-t1DEt=K0Wj+mFvkA_y_OFeTNy=G+iZ6pwqhd62iGm0KA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAFiTN-t1DEt=K0Wj+mFvkA_y_OFeTNy=G+iZ6pwqhd62iGm0KA@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk

On Mon, Mar 01, 2021 at 10:32:23AM +0530, Dilip Kumar wrote:
> On Sun, Feb 28, 2021 at 9:48 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> >
> > On my PC, this new change is causing a test failure:
> >
> >  SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
> > -                       substr
> > -----------------------------------------------------
> > - 01234567890123456789012345678901234567890123456789
> > -(1 row)
> > -
> > +ERROR:  compressed lz4 data is corrupt
> 
> The older version of lz4 had this problem that while decompressing
> partial if we don't give the buffer size up to full data length it was
> failing[1] but it is solved in version 1.9.

Thanks.  It seems like that explains it.                                                                                                                                                                                           
I think if that's a problem with recent versions, then you'll have to
conditionally disable slicing.
https://packages.debian.org/liblz4-dev                                                                                                                                                                                             

Slicing isn't generally usable if it sometimes makes people's data inaccessible
and gives errors about corruption.

I guess you could make it a compile time test on these constants (I don't know
the necessary version, though)

#define LZ4_VERSION_MAJOR    1    /* for breaking interface changes  */
#define LZ4_VERSION_MINOR    7    /* for new (non-breaking) interface capabilities */
#define LZ4_VERSION_RELEASE  1    /* for tweaks, bug-fixes, or development */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)

If the version is too low, either make it #error, or disable slicing.                                                                                                                                                              
The OS usual library version infrastructure will make sure the runtime version
is at least the MAJOR+MINOR of the compile time version.

-- 
Justin