From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Ashwin Agrawal <aagrawal@pivotal.io>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Cc: PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Date: Mon, 15 Apr 2019 22:17:09 +0200
Message-ID: <20190415201709.iuekkfen4df54pbg@development> (raw)
In-Reply-To: <CALfoeitpQ-yuZgdW=tzN7OpfzVS4uJEY6rszg9T71wS60Y07GA@mail.gmail.com>
References: <f301066e-08e1-a4b6-e058-0c1f24a1cc92@catalyst.net.nz>
<6A2B85A8-1ABD-4F85-981E-C94172D662FD@pivotal.io>
<11257.1554994460@sss.pgh.pa.us>
<d4bfe7b0-0836-06b1-6b03-f5874b5a2b00@iki.fi>
<20190414163947.owdw4gyjui36rakf@development>
<CALfoeiuc_FFXO00qZgEVsahw4_R5OKNWbc4+VWMxUfWm=iSeMg@mail.gmail.com>
<20190415173254.nlnk2xqhgt7c5pta@development>
<CALfoeitUdo62e-HTYqBxeBb+d0hW4+uXWggdWh5t_pbYTcE9Ng@mail.gmail.com>
<20190415181754.6dfp5lr6obpzutub@development>
<CALfoeitpQ-yuZgdW=tzN7OpfzVS4uJEY6rszg9T71wS60Y07GA@mail.gmail.com>
On Mon, Apr 15, 2019 at 11:57:49AM -0700, Ashwin Agrawal wrote:
> On Mon, Apr 15, 2019 at 11:18 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
>
> Maybe. I'm not going to pretend I fully understand the internals. Does
> that mean the container contains ZSUncompressedBtreeItem as elements? Or
> just the plain Datum values?
>
> First, your reading of code and all the comments/questions so far have
> been highly encouraging. Thanks a lot for the same.
;-)
> Container contains ZSUncompressedBtreeItem as elements. As for Item will
> have to store meta-data like size, undo and such info. We don't wish to
> restrict compressing only items from same insertion sessions only. Hence,
> yes doens't just store Datum values. Wish to consider it more tuple level
> operations and have meta-data for it and able to work with tuple level
> granularity than block level.
OK, thanks for the clarification, that somewhat explains my confusion.
So if I understand it correctly, ZSCompressedBtreeItem is essentially a
sequence of ZSUncompressedBtreeItem(s) stored one after another, along
with some additional top-level metadata.
> Definitely many more tricks can be and need to be applied to optimize
> storage format, like for fixed width columns no need to store the size in
> every item. Keep it simple is theme have been trying to maintain.
> Compression ideally should compress duplicate data pretty easily and
> efficiently as well, but we will try to optimize as much we can without
> the same.
I think there's plenty of room for improvement. The main problem I see
is that it mixes different types of data, which is bad for compression
and vectorized execution. I think we'll end up with a very different
representation of the container, essentially decomposing the items into
arrays of values of the same type - array of TIDs, array of undo
pointers, buffer of serialized values, etc.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, aagrawal@pivotal.io, hlinnaka@iki.fi, tgl@sss.pgh.pa.us, mark.kirkwood@catalyst.net.nz
Subject: Re: Zedstore - compressed in-core columnar storage
In-Reply-To: <20190415201709.iuekkfen4df54pbg@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox