From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Ashwin Agrawal <aagrawal@pivotal.io>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Cc: PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Date: Mon, 15 Apr 2019 20:17:54 +0200
Message-ID: <20190415181754.6dfp5lr6obpzutub@development> (raw)
In-Reply-To: <CALfoeitUdo62e-HTYqBxeBb+d0hW4+uXWggdWh5t_pbYTcE9Ng@mail.gmail.com>
References: <CALfoeiuF-m5jg51mJUPm5GN8u396o5sA2AF5N97vTRAEDYac7w@mail.gmail.com>
<fc2d4f99-1ca6-854f-7025-ec3db94f6e04@catalyst.net.nz>
<f301066e-08e1-a4b6-e058-0c1f24a1cc92@catalyst.net.nz>
<6A2B85A8-1ABD-4F85-981E-C94172D662FD@pivotal.io>
<11257.1554994460@sss.pgh.pa.us>
<d4bfe7b0-0836-06b1-6b03-f5874b5a2b00@iki.fi>
<20190414163947.owdw4gyjui36rakf@development>
<CALfoeiuc_FFXO00qZgEVsahw4_R5OKNWbc4+VWMxUfWm=iSeMg@mail.gmail.com>
<20190415173254.nlnk2xqhgt7c5pta@development>
<CALfoeitUdo62e-HTYqBxeBb+d0hW4+uXWggdWh5t_pbYTcE9Ng@mail.gmail.com>
On Mon, Apr 15, 2019 at 10:50:21AM -0700, Ashwin Agrawal wrote:
> On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
>
> ...
>
> I see. Perhaps it'd be better to call the flag ZSBT_CONTAINER, when it
> means "this is a container". And then have another flag to track whether
> the container is compressed or not. But as I suggested elsewhere in this
> thread, I think it might be better to store some ID of the compression
> algorithm used instead of a simple flag.
>
> FWIW when I had to deal with incremental compression (adding data into
> already compressed buffers), which is what seems to be happening here, I
> found it very useful/efficient to allow partially compressed buffers and
> only trigger recompressin when absolutely needed.
>
> Applied to this case, the container would first store compressed chunk,
> followed by raw (uncompressed) data. Say, like this:
>
> ZSContainerData {
>
> // header etc.
>
> int nbytes; /* total bytes in data */
> int ncompressed; /* ncompressed <= nbytes, fully compressed when
> * (ncompressed == nbytes) */
>
> char data[FLEXIBLE_ARRAY_MEMBER];
> }
>
> When adding a value to the buffer, it'd be simply appended to the data
> array. When the container would grow too much (can't fit on the page or
> something), recompression is triggered.
>
> I think what you suggested here is exactly how its handled currently, just
> the mechanics are little different. Plain items are added to page as
> insertions are performed. Then when page becomes full, compression is
> triggerred container item is created for them to store the compressed
> data. Then new insertions are stored as plain items, once again when page
> becomes full, they are compressed and container item created for it. So,
> never, compressed data is attempted to be compressed again. So, on page
> plain items are acting as data section you mentioned above. A page can
> have mix of n plain and n container items.
Maybe. I'm not going to pretend I fully understand the internals. Does
that mean the container contains ZSUncompressedBtreeItem as elements? Or
just the plain Datum values?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, aagrawal@pivotal.io, hlinnaka@iki.fi, tgl@sss.pgh.pa.us, mark.kirkwood@catalyst.net.nz
Subject: Re: Zedstore - compressed in-core columnar storage
In-Reply-To: <20190415181754.6dfp5lr6obpzutub@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox