Date: Mon, 15 Apr 2019 20:17:54 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Ashwin Agrawal <aagrawal@pivotal.io>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>, Tom Lane <tgl@sss.pgh.pa.us>,
	Mark Kirkwood <mark.kirkwood@catalyst.net.nz>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Zedstore - compressed in-core columnar storage
Message-ID: <20190415181754.6dfp5lr6obpzutub@development>
References: 
 <CALfoeiuF-m5jg51mJUPm5GN8u396o5sA2AF5N97vTRAEDYac7w@mail.gmail.com>
 <fc2d4f99-1ca6-854f-7025-ec3db94f6e04@catalyst.net.nz>
 <f301066e-08e1-a4b6-e058-0c1f24a1cc92@catalyst.net.nz>
 <6A2B85A8-1ABD-4F85-981E-C94172D662FD@pivotal.io>
 <11257.1554994460@sss.pgh.pa.us>
 <d4bfe7b0-0836-06b1-6b03-f5874b5a2b00@iki.fi>
 <20190414163947.owdw4gyjui36rakf@development>
 <CALfoeiuc_FFXO00qZgEVsahw4_R5OKNWbc4+VWMxUfWm=iSeMg@mail.gmail.com>
 <20190415173254.nlnk2xqhgt7c5pta@development>
 <CALfoeitUdo62e-HTYqBxeBb+d0hW4+uXWggdWh5t_pbYTcE9Ng@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
 <CALfoeitUdo62e-HTYqBxeBb+d0hW4+uXWggdWh5t_pbYTcE9Ng@mail.gmail.com>
User-Agent: NeoMutt/20180716-1444-295967
Precedence: bulk

On Mon, Apr 15, 2019 at 10:50:21AM -0700, Ashwin Agrawal wrote:
>   On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra
>   <tomas.vondra@2ndquadrant.com> wrote:
>
>     ...
>
>     I see. Perhaps it'd be better to call the flag ZSBT_CONTAINER, when it
>     means "this is a container". And then have another flag to track whether
>     the container is compressed or not. But as I suggested elsewhere in this
>     thread, I think it might be better to store some ID of the compression
>     algorithm used instead of a simple flag.
>
>     FWIW when I had to deal with incremental compression (adding data into
>     already compressed buffers), which is what seems to be happening here, I
>     found it very useful/efficient to allow partially compressed buffers and
>     only trigger recompressin when absolutely needed.
>
>     Applied to this case, the container would first store compressed chunk,
>     followed by raw (uncompressed) data. Say, like this:
>
>     ZSContainerData {
>
>         // header etc.
>
>         int nbytes;         /* total bytes in data */
>         int ncompressed;    /* ncompressed <= nbytes, fully compressed when
>                              * (ncompressed == nbytes) */
>
>         char data[FLEXIBLE_ARRAY_MEMBER];
>     }
>
>     When adding a value to the buffer, it'd be simply appended to the data
>     array. When the container would grow too much (can't fit on the page or
>     something), recompression is triggered.
>
>   I think what you suggested here is exactly how its handled currently, just
>   the mechanics are little different. Plain items are added to page as
>   insertions are performed. Then when page becomes full, compression is
>   triggerred container item is created for them to store the compressed
>   data. Then new insertions are stored as plain items, once again when page
>   becomes full, they are compressed and container item created for it. So,
>   never, compressed data is attempted to be compressed again. So, on page
>   plain items are acting as data section you mentioned above. A page can
>   have mix of n plain and n container items.

Maybe. I'm not going to pretend I fully understand the internals. Does
that mean the container contains ZSUncompressedBtreeItem as elements? Or
just the plain Datum values?

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services