Date: Sun, 14 Apr 2019 18:36:18 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Andreas Karlsson <andreas@proxel.se>,
	Ashwin Agrawal <aagrawal@pivotal.io>,
	Mark Kirkwood <mark.kirkwood@catalyst.net.nz>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Message-ID: <20190414163618.x2bhtfwdulhah7o5@development>
References: 
 <CALfoeiuF-m5jg51mJUPm5GN8u396o5sA2AF5N97vTRAEDYac7w@mail.gmail.com>
 <fc2d4f99-1ca6-854f-7025-ec3db94f6e04@catalyst.net.nz>
 <f301066e-08e1-a4b6-e058-0c1f24a1cc92@catalyst.net.nz>
 <6A2B85A8-1ABD-4F85-981E-C94172D662FD@pivotal.io>
 <44cafbdf-0a04-64ef-3cf8-6d5b14b68643@postgrespro.ru>
 <93222d26-2141-acf2-61ef-3127f57e80e6@proxel.se>
 <59ef6173-6667-8a02-ff7c-a52c8ce05a4f@postgrespro.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <59ef6173-6667-8a02-ff7c-a52c8ce05a4f@postgrespro.ru>
User-Agent: NeoMutt/20180716-1444-295967
Precedence: bulk

On Thu, Apr 11, 2019 at 04:52:33PM +0300, Konstantin Knizhnik wrote:
>   On 11.04.2019 16:18, Andreas Karlsson wrote:
>
>     On 4/11/19 10:46 AM, Konstantin Knizhnik wrote:
>
>       This my results of compressing pbench data using different
>       compressors:
>
>       +-------------------------------------------------------------+
>       |Configuration              |Size (Gb)       |Time (sec)      |
>       |---------------------------+----------------+----------------|
>       |no compression             |15.31           |92              |
>       |---------------------------+----------------+----------------|
>       |zlib (default level)       |2.37            |284             |
>       |---------------------------+----------------+----------------|
>       |zlib (best speed)          |2.43            |191             |
>       |---------------------------+----------------+----------------|
>       |postgres internal lz       |3.89            |214             |
>       |---------------------------+----------------+----------------|
>       |lz4                        |4.12            |95              |
>       |---------------------------+----------------+----------------|
>       |snappy                     |5.18            |99              |
>       |---------------------------+----------------+----------------|
>       |lzfse                      |2.80            |1099            |
>       |---------------------------+----------------+----------------|
>       |(apple) 2.80 1099          |1.69            |125             |
>       +-------------------------------------------------------------+
>
>       You see that zstd provides almost 2 times better compression ration
>       and almost at the same speed.
>
>     What is "(apple) 2.80 1099"? Was that intended to be zstd?
>
>     Andreas
>
>   Ugh...
>   Cut and paste problems.
>   The whole document can be found here:
>   http://garret.ru/PageLevelCompression.pdf
>
>   lzfse (apple)       2.80    1099
>   zstd (facebook)  1.69    125
>
>   ztsd is compression algorithm proposed by facebook: 
>   https://github.com/facebook/zstd
>   Looks like it provides the best speed/compress ratio result.
>

I think those comparisons are cute and we did a fair amount of them when
considering a drop-in replacement for pglz, but ultimately it might be a
bit pointless because:

(a) it very much depends on the dataset (one algorithm may work great on
one type of data, suck on another)

(b) different systems may require different trade-offs (high ingestion
rate vs. best compression ratio)

(c) decompression speed may be much more important

What I'm trying to say is that we shouldn't obsess about picking one
particular algorithm too much, because it's entirely pointless. Instead,
we should probably design the system to support different compression
algorithms, ideally at column level.

Also, while these general purpose algorithms are nice, what I think will
be important in later stages of colstore development will be compression
algorithms allowing execution directly on the compressed data (like RLE,
dictionary and similar approaches).


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services