Date: Tue, 21 Nov 2017 23:28:55 +0300
From: Ildus K <i.kurbangaliev@postgrespro.ru>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Custom compression methods
Message-ID: <20171121232855.349e14a9@postgrespro.ru>
In-Reply-To: <bfb3d34e-ba74-02cb-ca2a-43344866d9fd@2ndquadrant.com>
References: <20170907194236.4cefce96@wp.localdomain>
 <20170912175505.4afa11fd@wp.localdomain>
 <20171102152836.60c041e4@wp.localdomain>
 <20171114162356.52e3d388@wp.localdomain>
 <62e46a47-08a8-6e06-5e4f-6f52e0a13202@2ndquadrant.com>
 <20171121174717.69ecd8f4@wp.localdomain>
 <bfb3d34e-ba74-02cb-ca2a-43344866d9fd@2ndquadrant.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Tue, 21 Nov 2017 18:47:49 +0100
Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:

 
> 
> I propose to use either
> 
>    CompressionMethodOptions (and CompressionMethodRoutine)
> 
> or
> 
>    CompressionOptions (and CompressionRoutine)

Sounds good, thanks.

> 
> OK. But then I don't understand why tsvector.c does things like
> 
>     VARSIZE(data) - VARHDRSZ_CUSTOM_COMPRESSED - arrsize
>     VARRAWSIZE_4B_C(data) - arrsize
> 
> instead of
> 
>     VARSIZE_ANY_EXHDR(data) - arrsize
>     VARSIZE_ANY(data) - arrsize
> 
> Seems somewhat confusing.
> 

VARRAWSIZE_4B_C returns original size of data, before compression (from
va_rawsize in current postgres, and from va_info in my patch), not size
of the already compressed data, so you can't use VARSIZE_ANY here.

VARSIZE_ANY_EXHDR in current postgres returns VARSIZE-VARHDRSZ, despite
the varlena is compressed or not, so I just kept this behavior for
custom compressed varlenas too. If you look into tuptoaster.c you will
also see lines like 'VARSIZE(attr) - TOAST_COMPRESS_HDRSZ'. So I think
if VARSIZE_ANY_EXHDR will subtract different header sizes then it
should subtract them for usual compressed varlenas too.

> >   
> 
> Hmmm, it still doesn't work for me. See this:
> 
>     test=# create extension pg_lz4 ;
>     CREATE EXTENSION
>     test=# create table t_lz4 (v text compressed lz4);
>     CREATE TABLE
>     test=# create table t_pglz (v text);
>     CREATE TABLE
>     test=# insert into t_lz4 select repeat(md5(1::text),300);
>     INSERT 0 1
>     test=# insert into t_pglz select * from t_lz4;
>     INSERT 0 1
>     test=# drop extension pg_lz4 cascade;
>     NOTICE:  drop cascades to 2 other objects
>     DETAIL:  drop cascades to compression options for lz4
>     drop cascades to table t_lz4 column v
>     DROP EXTENSION
>     test=# \c test
>     You are now connected to database "test" as user "user".
>     test=# insert into t_lz4 select repeat(md5(1::text),300);^C
>     test=# select * from t_pglz ;
>     ERROR:  cache lookup failed for compression options 16419
> 
> That suggests no recompression happened.

I will check that. Is your extension published somewhere?