Subject: Re: [HACKERS] Custom compression methods
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>,
 Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>,
 "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>,
 Ildar Musin <i.musin@postgrespro.ru>
References: <9a0745ab-5b62-d81f-3a6d-ec0929a4a99d@2ndquadrant.com>
 <20171130182009.1b492eb2@wp.localdomain>
 <9dc0d3ba-5fa2-a7b0-5789-fff3c60cef1c@2ndquadrant.com>
 <CA+TgmoZFEhqxXBOONMQeCZ09AJGqE=H2UMySF4zHZsatA0_E2Q@mail.gmail.com>
 <a6fe2ee1-7f0e-67d1-7c5d-5075c17191d6@2ndquadrant.com>
 <CA+TgmoYGvNV7GqKLLCYcdFEhiANHA_mXxtf_EB=Pd--y14sMQg@mail.gmail.com>
 <b6a47ebe-35e3-ad1e-64d9-6e834708ec5a@2ndquadrant.com>
 <CA+TgmoZY+TB-wa6QgwsaL-MHCJAhR9n0ifEKTDq8PTxBuqEW-w@mail.gmail.com>
 <20171201215217.7ic4izdvj5cishgx@alap3.anarazel.de>
 <5cb1995e-a202-2739-bf46-1b6950e5bdb6@2ndquadrant.com>
 <20171202203839.lvbsfyu73igrdmsu@alap3.anarazel.de>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <c1b3880a-7a4c-c5bd-7be8-beec0c8684b8@2ndquadrant.com>
Date: Sat, 2 Dec 2017 22:53:10 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.3.0
MIME-Version: 1.0
In-Reply-To: <20171202203839.lvbsfyu73igrdmsu@alap3.anarazel.de>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 12/02/2017 09:38 PM, Andres Freund wrote:
> Hi,
> 
> On 2017-12-02 16:04:52 +0100, Tomas Vondra wrote:
>> Firstly, it's going to be quite hard (or perhaps impossible) to find an
>> algorithm that is "universally better" than pglz. Some algorithms do
>> work better for text documents, some for binary blobs, etc. I don't
>> think there's a win-win option.
> 
> lz4 is pretty much there.
> 

That's a matter of opinion, I guess. It's a solid compression algorithm,
that's for sure ...

>> Secondly, all the previous attempts ran into some legal issues, i.e.
>> licensing and/or patents. Maybe the situation changed since then (no
>> idea, haven't looked into that), but in the past the "pluggable"
>> approach was proposed as a way to address this.
> 
> Those were pretty bogus.

IANAL so I don't dare to judge on bogusness of such claims. I assume if
we made it optional (e.g. configure/initdb option, it'd be much less of
an issue). Of course, that has disadvantages too (because when you
compile/init with one algorithm, and then find something else would work
better for your data, you have to start from scratch).

>
> I think we're not doing our users a favor if they've to download
> some external projects, then fiddle with things, just to not choose
> a compression algorithm that's been known bad for at least 5+ years.
> If we've a decent algorithm in-core *and* then allow extensibility, 
> that's one thing, but keeping the bad and tell forks "please take
> our users with this code we give you" is ...
> 

I don't understand what exactly is your issue with external projects,
TBH. I think extensibility is one of the great strengths of Postgres.
It's not all rainbows and unicorns, of course, and it has costs too.

FWIW I don't think pglz is a "known bad" algorithm. Perhaps there are
cases where other algorithms (e.g. lz4) are running circles around it,
particularly when it comes to decompression speed, but I wouldn't say
it's "known bad".

Not sure which forks you're talking about ...

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services