Subject: Re: Zedstore - compressed in-core columnar storage
To: Heikki Linnakangas <hlinnaka@iki.fi>,
 Amit Kapila <amit.kapila16@gmail.com>, Ashwin Agrawal <aagrawal@pivotal.io>
Cc: PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
References: 
 <CALfoeiuF-m5jg51mJUPm5GN8u396o5sA2AF5N97vTRAEDYac7w@mail.gmail.com>
 <CAA4eK1Kw1h6SE9dO243T8cvak4CPQcvEXpJMPBzkGdVwRP_tBg@mail.gmail.com>
 <7ab5be5f-6404-b52b-e6bf-c1c891ceabb1@iki.fi>
From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Message-ID: <b522e7a7-e7f4-47f1-7430-c594f3f712b6@postgrespro.ru>
Date: Wed, 10 Apr 2019 10:38:13 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
In-Reply-To: <7ab5be5f-6404-b52b-e6bf-c1c891ceabb1@iki.fi>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk


On 10.04.2019 10:25, Heikki Linnakangas wrote:
> On 10/04/2019 09:29, Amit Kapila wrote:
>> On Tue, Apr 9, 2019 at 5:57 AM Ashwin Agrawal <aagrawal@pivotal.io> 
>> wrote:
>>> Row store
>>> ---------
>>>
>>> The tuples are stored one after another, sorted by TID. For each
>>> tuple, we store its 48-bit TID, a undo record pointer, and the actual
>>> tuple data uncompressed.
>>>
>>
>> Storing undo record pointer with each tuple can take quite a lot of
>> space in cases where you can't compress them.
>
> Yeah. This does depend on compression to eliminate the unused fields 
> quite heavily at the moment. But you could have a flag in the header 
> to indicate "no undo pointer needed", and just leave it out, when it's 
> needed.
>
>> Have you thought how will you implement the multi-locker scheme in
>> this design?  In zheap, we have used undo for the same and it is easy
>> to imagine when you have separate transaction slots for each
>> transaction.  I am not sure how will you implement the same here.
> I've been thinking that the undo record would store all the XIDs 
> involved. So if there are multiple lockers, the UNDO record would 
> store a list of XIDs. Alternatively, I suppose you could store 
> multiple UNDO pointers for the same tuple.
>
> - Heikki
>
>

I also a little bit confused about UNDO records and MVCC support in 
Zedstore. Actually columnar store is mostly needed for analytic for
read-only or append-only data. One of the disadvantages of Postgres is 
quite larger per-record space overhead caused by MVCC.
It may be critical if you want to store huge timeseries  with relatively 
small number of columns (like measurements of some sensor).
It will be nice to have storage format which reduce this overhead when 
it is not needed (data is not updated).

Right now, even without UNFO pages, size of zedstore is larger than size 
of original Postgres table.
It seems to be very strange.


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company