From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Vladimir Leskov <vladimirlesk@yandex-team.ru>
Subject: Re: pglz performance
Date: Fri, 2 Aug 2019 19:00:39 +0200
Message-ID: <20190802170039.o4pabnzm4xy3z7uj@development> (raw)
In-Reply-To: <20190802163948.i6mjypdgujeorrbi@alap3.anarazel.de>
References: <469C9ED9-348C-4FE7-A7A7-B0FA671BEE4C@yandex-team.ru>
<20190513071427.GB2273@paquier.xyz>
<169163A8-C96F-4DBE-A062-7D1CECBE9E5D@yandex-team.ru>
<ea57b49a-ecf0-481a-a77b-631833354f7d@postgrespro.ru>
<20190802144345.d62jtiyyx6r2y73f@development>
<DBB2A9E5-29FD-40BF-AC60-BD990FBF142F@yandex-team.ru>
<20190802163948.i6mjypdgujeorrbi@alap3.anarazel.de>
On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
>> We have some kind of "roadmap" of "extensible pglz". We plan to
>> provide implementation on Novembers CF.
>
>I don't understand why it's a good idea to improve the compression side
>of pglz. There's plenty other people that spent a lot of time
>developing better compression algorithms.
>
Isn't it beneficial for existing systems, that will be stuck with pglz
even if we end up adding other algorithms?
>
>> Currently, pglz starts with empty cache map: there is no prior 4k
>> bytes before start. We can add imaginary prefix to any data with
>> common substrings: this will enhance compression ratio. It is hard
>> to decide on training data set for this "common prefix". So we want
>> to produce extension with aggregate function which produces some
>> "adapted common prefix" from users's data. Then we can "reserve" few
>> negative bytes for "decompression commands". This command can
>> instruct database on which common prefix to use. But also system
>> command can say "invoke decompression from extension".
>>
>> Thus, user will be able to train database compression on his data and
>> substitute pglz compression with custom compression method
>> seamlessly.
>>
>> This will make hard-choosen compression unneeded, but seems overly
>> hacky. But there will be no need to have lz4, zstd, brotli, lzma and
>> others in core. Why not provide e.g. "time series compression"? Or
>> "DNA compression"? Whatever gun user wants for his foot.
>
>I think this is way too complicated, and will provide not particularly
>much benefit for the majority users.
>
I agree with this. I do see value in the feature, but probably not as a
drop-in replacement for the default compression algorithm. I'd compare
it to the "custom compression methods" patch that was submitted some
time ago.
>In fact, I'll argue that we should flat out reject any such patch until
>we have at least one decent default compression algorithm in core.
>You're trying to work around a poor compression algorithm with
>complicated dictionary improvement, that require user interaction, and
>only will work in a relatively small subset of the cases, and will very
>often increase compression times.
>
I wouldn't be so strict I guess. But I do agree an algorithm that
requires additional steps (training, ...) is unlikely to be a good
candidate for default instance compression alorithm.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, andres@anarazel.de, x4mmm@yandex-team.ru, k.knizhnik@postgrespro.ru, michael@paquier.xyz, vladimirlesk@yandex-team.ru
Subject: Re: pglz performance
In-Reply-To: <20190802170039.o4pabnzm4xy3z7uj@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox