From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Andreas Karlsson <andreas@proxel.se>
Cc: Ashwin Agrawal <aagrawal@pivotal.io>
Cc: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Cc: PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Date: Sun, 14 Apr 2019 18:36:18 +0200
Message-ID: <20190414163618.x2bhtfwdulhah7o5@development> (raw)
In-Reply-To: <59ef6173-6667-8a02-ff7c-a52c8ce05a4f@postgrespro.ru>
References: <CALfoeiuF-m5jg51mJUPm5GN8u396o5sA2AF5N97vTRAEDYac7w@mail.gmail.com>
<fc2d4f99-1ca6-854f-7025-ec3db94f6e04@catalyst.net.nz>
<f301066e-08e1-a4b6-e058-0c1f24a1cc92@catalyst.net.nz>
<6A2B85A8-1ABD-4F85-981E-C94172D662FD@pivotal.io>
<44cafbdf-0a04-64ef-3cf8-6d5b14b68643@postgrespro.ru>
<93222d26-2141-acf2-61ef-3127f57e80e6@proxel.se>
<59ef6173-6667-8a02-ff7c-a52c8ce05a4f@postgrespro.ru>
On Thu, Apr 11, 2019 at 04:52:33PM +0300, Konstantin Knizhnik wrote:
> On 11.04.2019 16:18, Andreas Karlsson wrote:
>
> On 4/11/19 10:46 AM, Konstantin Knizhnik wrote:
>
> This my results of compressing pbench data using different
> compressors:
>
> +-------------------------------------------------------------+
> |Configuration |Size (Gb) |Time (sec) |
> |---------------------------+----------------+----------------|
> |no compression |15.31 |92 |
> |---------------------------+----------------+----------------|
> |zlib (default level) |2.37 |284 |
> |---------------------------+----------------+----------------|
> |zlib (best speed) |2.43 |191 |
> |---------------------------+----------------+----------------|
> |postgres internal lz |3.89 |214 |
> |---------------------------+----------------+----------------|
> |lz4 |4.12 |95 |
> |---------------------------+----------------+----------------|
> |snappy |5.18 |99 |
> |---------------------------+----------------+----------------|
> |lzfse |2.80 |1099 |
> |---------------------------+----------------+----------------|
> |(apple) 2.80 1099 |1.69 |125 |
> +-------------------------------------------------------------+
>
> You see that zstd provides almost 2 times better compression ration
> and almost at the same speed.
>
> What is "(apple) 2.80 1099"? Was that intended to be zstd?
>
> Andreas
>
> Ugh...
> Cut and paste problems.
> The whole document can be found here:
> http://garret.ru/PageLevelCompression.pdf
>
> lzfse (apple) 2.80 1099
> zstd (facebook) 1.69 125
>
> ztsd is compression algorithm proposed by facebook:
> https://github.com/facebook/zstd
> Looks like it provides the best speed/compress ratio result.
>
I think those comparisons are cute and we did a fair amount of them when
considering a drop-in replacement for pglz, but ultimately it might be a
bit pointless because:
(a) it very much depends on the dataset (one algorithm may work great on
one type of data, suck on another)
(b) different systems may require different trade-offs (high ingestion
rate vs. best compression ratio)
(c) decompression speed may be much more important
What I'm trying to say is that we shouldn't obsess about picking one
particular algorithm too much, because it's entirely pointless. Instead,
we should probably design the system to support different compression
algorithms, ideally at column level.
Also, while these general purpose algorithms are nice, what I think will
be important in later stages of colstore development will be compression
algorithms allowing execution directly on the compressed data (like RLE,
dictionary and similar approaches).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, k.knizhnik@postgrespro.ru, andreas@proxel.se, aagrawal@pivotal.io, mark.kirkwood@catalyst.net.nz
Subject: Re: Zedstore - compressed in-core columnar storage
In-Reply-To: <20190414163618.x2bhtfwdulhah7o5@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox