Subject: Re: libpq compression
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>,
 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
 Robbie Harwood <rharwood@redhat.com>,
 Thomas Munro <thomas.munro@enterprisedb.com>
Cc: Grigory Smolkin <g.smolkin@postgrespro.ru>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <aad16e41-b3f9-e89d-fa57-fb4c694bec25@postgrespro.ru>
 <dbf9035f-7f7b-b053-cc2a-52e5147d4557@postgrespro.ru>
 <429cf55b-a8bf-c4ba-196d-5b45420145a1@postgrespro.ru>
 <CAEepm=3CGUX4Vyw=jFaOedt99HS1HQiBKYTX4OYHo7HU2AczZg@mail.gmail.com>
 <36db5d02-45e6-b863-1418-825ef20ae4c6@postgrespro.ru>
 <CAEepm=3RAS8xVQK8rs+Xitzp+JXzGdGsMA7f+H8fjKQ7Do8aSA@mail.gmail.com>
 <f4ad0809-dd90-e321-5c1a-0ca68a136a8f@postgrespro.ru>
 <jlg4lhzdeak.fsf@redhat.com>
 <3b0260da-fe47-2341-da01-51383fe9172d@postgrespro.ru>
 <ebc8ebd2-1a3b-43f7-d1a1-6e8b9ed65e62@2ndquadrant.com>
 <cd974e7e-85da-19a4-f25b-a7a56ff87d00@postgrespro.ru>
 <739641a6-f64d-0f2b-8483-34ab23e2dcd6@2ndquadrant.com>
From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Message-ID: <d58ebe47-5662-a50f-117d-4c1049acbfef@postgrespro.ru>
Date: Fri, 15 Feb 2019 18:40:37 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
In-Reply-To: <739641a6-f64d-0f2b-8483-34ab23e2dcd6@2ndquadrant.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk


On 15.02.2019 18:26, Tomas Vondra wrote:
>
>
> On 2/15/19 3:03 PM, Konstantin Knizhnik wrote:
>>
>> On 15.02.2019 15:42, Peter Eisentraut wrote:
>>> On 2018-06-19 09:54, Konstantin Knizhnik wrote:
>>>> The main drawback of streaming compression is that you can not
>>>> decompress some particular message without decompression of all previous
>>>> messages.
>>> It seems this would have an adverse effect on protocol-aware connection
>>> proxies: They would have to uncompress everything coming in and
>>> recompress everything going out.
>>>
>>> The alternative of compressing each packet individually would work much
>>> better: A connection proxy could peek into the packet header and only
>>> uncompress the (few, small) packets that it needs for state and routing.
>>>
>> Individual compression of each message depreciate all idea of libpq
>> compression. Messages are two small to efficiently compress each of
>> them separately. So using streaming compression algorithm is
>> absolutely necessary here.
>>
> Hmmm, I see Peter was talking about "packets" while you're talking about
> "messages". Are you talking about the same thing?
Sorry, but there are no "packet" in libpq protocol, so I assumed that 
packet=message.
In any protocol-aware proxy has to proceed each message.

>
> Anyway, I was going to write about the same thing - that per-message
> compression would likely eliminate most of the benefits - but I'm
> wondering if it's actually true. That is, how much will the compression
> ratio drop if we compress individual messages?
Compression of small messages without shared dictionary will give awful 
results.
Assume that average record and so message size is 100 bytes.
Just perform very simple experiment create file with 100 equal 
characters and try to compress it.
With zlib result will be 173 bytes. So after "compression" size of file 
is increase 1.7 times.
This is why there is no other way to efficiently compress libpq traffic 
without usage of streaming compression
(when dictionary is  shared and updated for all messages).
>
> Obviously, if there are just tiny messages, it might easily eliminate
> any benefits (and in fact it would add overhead). But I'd say we're way
> more interested in transferring large data sets (result sets, data for
> copy, etc.) and presumably those messages are much larger. So maybe we
> could compress just those, somehow?
Please notice that copy stream consists of individual messages for each 
record.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company