Date: Sun, 14 Apr 2019 19:12:33 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>,
	Andreas Karlsson <andreas@proxel.se>,
	Ashwin Agrawal <aagrawal@pivotal.io>,
	Mark Kirkwood <mark.kirkwood@catalyst.net.nz>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Message-ID: <20190414171233.vhukbvlfwnacc37u@development>
References: 
 <CALfoeiuF-m5jg51mJUPm5GN8u396o5sA2AF5N97vTRAEDYac7w@mail.gmail.com>
 <fc2d4f99-1ca6-854f-7025-ec3db94f6e04@catalyst.net.nz>
 <f301066e-08e1-a4b6-e058-0c1f24a1cc92@catalyst.net.nz>
 <6A2B85A8-1ABD-4F85-981E-C94172D662FD@pivotal.io>
 <44cafbdf-0a04-64ef-3cf8-6d5b14b68643@postgrespro.ru>
 <93222d26-2141-acf2-61ef-3127f57e80e6@proxel.se>
 <59ef6173-6667-8a02-ff7c-a52c8ce05a4f@postgrespro.ru>
 <20190414163618.x2bhtfwdulhah7o5@development>
 <20190414164510.idlavfnqkutj2h3u@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <20190414164510.idlavfnqkutj2h3u@alap3.anarazel.de>
User-Agent: NeoMutt/20180716-1444-295967
Precedence: bulk

On Sun, Apr 14, 2019 at 09:45:10AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-04-14 18:36:18 +0200, Tomas Vondra wrote:
>> I think those comparisons are cute and we did a fair amount of them when
>> considering a drop-in replacement for pglz, but ultimately it might be a
>> bit pointless because:
>>
>> (a) it very much depends on the dataset (one algorithm may work great on
>> one type of data, suck on another)
>>
>> (b) different systems may require different trade-offs (high ingestion
>> rate vs. best compression ratio)
>>
>> (c) decompression speed may be much more important
>>
>> What I'm trying to say is that we shouldn't obsess about picking one
>> particular algorithm too much, because it's entirely pointless. Instead,
>> we should probably design the system to support different compression
>> algorithms, ideally at column level.
>
>I think we still need to pick a default algorithm, and realistically
>that's going to be used by like 95% of the users.
>

True. Do you expect it to be specific to the column store, or should be
set per-instance default (even for regular heap)?

FWIW I think the conclusion from past dev meetings was we're unlikely to
find anything better than lz4. I doubt that changed very much.

regard

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services