Subject: Re: pglz performance
To: Andres Freund <andres@anarazel.de>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>,
 Andrey Borodin <x4mmm@yandex-team.ru>,
 Konstantin Knizhnik <k.knizhnik@postgrespro.ru>,
 Michael Paquier <michael@paquier.xyz>,
 pgsql-hackers <pgsql-hackers@postgresql.org>,
 Vladimir Leskov <vladimirlesk@yandex-team.ru>
References: <ea57b49a-ecf0-481a-a77b-631833354f7d@postgrespro.ru>
 <20190802144345.d62jtiyyx6r2y73f@development>
 <DBB2A9E5-29FD-40BF-AC60-BD990FBF142F@yandex-team.ru>
 <20190802163948.i6mjypdgujeorrbi@alap3.anarazel.de>
 <20190802170039.o4pabnzm4xy3z7uj@development>
 <20190802171258.nob5zo2vu7siqo2b@alap3.anarazel.de>
 <20190802175239.5ds6rhdso6hwy4vk@development>
 <20190802182003.u477q6itwudieqxf@alap3.anarazel.de>
 <20190802194859.qxzxq2achzottkup@development>
 <d8576096-76ba-487d-515b-44fdedba8bb5@2ndquadrant.com>
 <20190804192034.smjcedlyjiusgtgg@alap3.anarazel.de>
From: Petr Jelinek <petr@2ndquadrant.com>
Message-ID: <4df61f6c-5800-30f0-7f84-cef4847e8a07@2ndquadrant.com>
Date: Mon, 5 Aug 2019 00:08:37 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <20190804192034.smjcedlyjiusgtgg@alap3.anarazel.de>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi,

On 04/08/2019 21:20, Andres Freund wrote:
> On 2019-08-04 02:41:24 +0200, Petr Jelinek wrote:
>> Same here.
>>
>> Just so that we don't idly talk, what do you think about the attached?
> 
> Cool!
> 
>> It:
>> - adds new GUC compression_algorithm with possible values of pglz (default)
>> and lz4 (if lz4 is compiled in), requires SIGHUP
> 
> As Tomas remarked, I think it shouldn't be SIGHUP but USERSET. And I
> think lz4 should be preferred, if available.  I could see us using a
> list style guc, so we could set it to lz4, pglz, and the first available
> one would be used.
> 

Sounds reasonable.

>> - adds 1 byte header to the compressed data where we currently store the
>> algorithm kind, that leaves us with 254 more to add :) (that's an extra
>> overhead compared to the current state)
> 
> Hm. Why do we need an additional byte?  IIRC my patch added that only
> for the case we would run out of space for compression formats without
> extending any sizes?
> 

Yeah your patch worked differently (I didn't actually use any code from 
it). The main reason why I add the byte is that I am storing the 
algorithm in the compressed value itself, not in varlena header. I was 
mainly trying to not have every caller care about storing and loading 
the compression algorithm. I also can't say I particularly like that 
hack in your patch.

However if we'd want to have separate GUCs for TOAST and WAL then we'll 
have to do that anyway so maybe it does not matter anymore (we can't use 
similar hack there AFAICS though).

> 
>> - changes the rawsize in TOAST header to 31 bits via bit packing
>> - uses the extra bit to differentiate between old and new format
> 
> Hm. Wouldn't it be easier to just use a different vartag for this?
> 

That would only work for external TOAST pointers right? The compressed 
varlena can also be stored inline and potentially in index tuple.

> 
>> - I expect my changes to configure.in are not the greatest as I don't have
>> pretty much zero experience with autoconf
> 
> FWIW the configure output changes are likely because you used a modified
> version of autoconf. Unfortunately debian/ubuntu ship one with vendor
> patches.
> 

Yeah, Ubuntu here, that explains.

-- 
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/