Date: Fri, 2 Aug 2019 19:52:39 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Andrey Borodin <x4mmm@yandex-team.ru>,
	Konstantin Knizhnik <k.knizhnik@postgrespro.ru>,
	Michael Paquier <michael@paquier.xyz>,
	pgsql-hackers <pgsql-hackers@postgresql.org>,
	Vladimir Leskov <vladimirlesk@yandex-team.ru>
Subject: Re: pglz performance
Message-ID: <20190802175239.5ds6rhdso6hwy4vk@development>
References: <469C9ED9-348C-4FE7-A7A7-B0FA671BEE4C@yandex-team.ru>
 <20190513071427.GB2273@paquier.xyz>
 <169163A8-C96F-4DBE-A062-7D1CECBE9E5D@yandex-team.ru>
 <ea57b49a-ecf0-481a-a77b-631833354f7d@postgrespro.ru>
 <20190802144345.d62jtiyyx6r2y73f@development>
 <DBB2A9E5-29FD-40BF-AC60-BD990FBF142F@yandex-team.ru>
 <20190802163948.i6mjypdgujeorrbi@alap3.anarazel.de>
 <20190802170039.o4pabnzm4xy3z7uj@development>
 <20190802171258.nob5zo2vu7siqo2b@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <20190802171258.nob5zo2vu7siqo2b@alap3.anarazel.de>
User-Agent: NeoMutt/20180716-1444-295967
Precedence: bulk

On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote:
>> On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote:
>> > Hi,
>> >
>> > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
>> > > We have some kind of "roadmap" of "extensible pglz". We plan to
>> > > provide implementation on Novembers CF.
>> >
>> > I don't understand why it's a good idea to improve the compression side
>> > of pglz. There's plenty other people that spent a lot of time
>> > developing better compression algorithms.
>> >
>>
>> Isn't it beneficial for existing systems, that will be stuck with pglz
>> even if we end up adding other algorithms?
>
>Why would they be stuck continuing to *compress* with pglz? As we
>fully retoast on write anyway we can just gradually switch over to the
>better algorithm. Decompression speed is another story, of course.
>

Hmmm, I don't remember the details of those patches so I didn't realize
it allows incremental recompression. If that's possible, that would mean 
existing systems can start using it. Which is good.

Another question is whether we'd actually want to include the code in
core directly, or use system libraries (and if some packagers might
decide to disable that, for whatever reason).

But yeah, I agree you may have a point about optimizing pglz compression.

>
>Here's what I had a few years back:
>
>https://www.postgresql.org/message-id/20130621000900.GA12425%40alap2.anarazel.de
>see also
>https://www.postgresql.org/message-id/20130605150144.GD28067%40alap2.anarazel.de
>
>I think we should refresh something like that patch, and:
>- make the compression algorithm GUC an enum, rename
>- add --with-system-lz4
>- obviously refresh the copy of lz4
>- drop snappy
>

That's a reasonable plan, I guess.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services