Date: Sun, 21 Mar 2021 18:43:24 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Dilip Kumar <dilipbalaut@gmail.com>,
	Robert Haas <robertmhaas@gmail.com>,
	Andres Freund <andres@anarazel.de>,
	Tomas Vondra <tomas.vondra@2ndquadrant.com>,
	Alexander Korotkov <a.korotkov@postgrespro.ru>,
	David Steele <david@pgmasters.net>,
	Ildus Kurbangaliev <i.kurbangaliev@gmail.com>,
	Dmitry Dolgov <9erthalion6@gmail.com>,
	Andrey Borodin <x4mmm@yandex-team.ru>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: [HACKERS] Custom compression methods (mac+lz4.h)
Message-ID: <20210321234324.GC4203@telsasoft.com>
References: <301730.1616276227@sss.pgh.pa.us>
 <20210320231957.GX11765@telsasoft.com>
 <309337.1616287493@sss.pgh.pa.us>
 <311370.1616290403@sss.pgh.pa.us>
 <CAFiTN-tN-fi=9VEatyiZyhW28yhhQCmn-egYzbdxBTLkOnVQvg@mail.gmail.com>
 <CAFiTN-t0w+Rc2U3S+y=7KWcLuOYNB5MfWeGdNa7+pg0UovVdcQ@mail.gmail.com>
 <414943.1616358751@sss.pgh.pa.us>
 <20210321204148.GB4203@telsasoft.com>
 <456616.1616360715@sss.pgh.pa.us>
 <504396.1616368310@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <504396.1616368310@sss.pgh.pa.us>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20210321234324.GC4203%40telsasoft.com>
Precedence: bulk

On Sun, Mar 21, 2021 at 07:11:50PM -0400, Tom Lane wrote:
> I wrote:
> > Justin Pryzby <pryzby@telsasoft.com> writes:
> >> On Sun, Mar 21, 2021 at 04:32:31PM -0400, Tom Lane wrote:
> >>> This seems somewhat repeatable (three identical failures in three
> >>> attempts).  Not sure why I did not see it yesterday; but anyway,
> >>> there is something wrong with partial detoasting for LZ4.
> 
> >> With what version of LZ4 ?
> 
> > RHEL8's, which is
> > lz4-1.8.3-2.el8.x86_64
> 
> I hate to be the bearer of bad news, but this suggests that
> LZ4_decompress_safe_partial is seriously broken in 1.9.2
> as well:
> 
> https://github.com/lz4/lz4/issues/783

Ouch

> Maybe we cannot rely on that function for a few more years yet.
> 
> Also, I don't really understand why this code:
> 
> 	/* slice decompression not supported prior to 1.8.3 */
> 	if (LZ4_versionNumber() < 10803)
> 		return lz4_decompress_datum(value);
> 
> It seems likely to me that we'd get a flat out build failure
> from library versions lacking LZ4_decompress_safe_partial,
> and thus that this run-time test is dead code and we should
> better be using a configure probe if we intend to allow old
> liblz4 versions.  Though that might be moot.

The function existed before 1.8.3, but didn't handle slicing.
https://github.com/lz4/lz4/releases/tag/v1.8.3
|Finally, an existing function, LZ4_decompress_safe_partial(), has been enhanced to make it possible to decompress only the beginning of an LZ4 block, up to a specified number of bytes. Partial decoding can be useful to save CPU time and memory, when the objective is to extract a limited portion from a larger block.

Possibly we could allow v >= 1.9.3 || (ver >= 1.8.3 && ver < 1.9.2).

Or maybe not: the second half apparently worked "by accident", and we shouldn't
need to have intimate knowledge of someone else's patchlevel releases, 

-- 
Justin