From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Andres Freund <andres@anarazel.de>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Даниил Захлыстов <usernamedt@yandex-team.ru>
Subject: Re: libpq compression
Date: Fri, 6 Nov 2020 09:28:03 +0300
Message-ID: <a811704f-0704-f0c7-e3f8-a44aa1f082cd@postgrespro.ru> (raw)
In-Reply-To: <5c832f37-db91-edec-3b31-7caef365df69@enterprisedb.com>
References: <c5421f2e-918a-2e74-f91a-a951d49012c4@postgrespro.ru>
<a112e0a2-6908-5a0d-ac22-012ebb6169b0@postgrespro.ru>
<74F0A385-6ED4-4EA7-9498-1A8527F16472@yandex-team.ru>
<a13ce4d1-768c-1449-25f6-0cd645499d93@postgrespro.ru>
<fa9113c0-db7a-2ed8-111c-a082a2217e4b@postgrespro.ru>
<22446ba0-058d-624a-a679-edc0215a56c8@postgrespro.ru>
<20201028192712.xj2qneyb3fvmjm6c@alap3.anarazel.de>
<3b1d2752-b7b5-db39-84c9-4ff024041fff@postgrespro.ru>
<20201030210308.54jjmnvhspk6p7cb@alap3.anarazel.de>
<059E5FF6-96A6-4DCE-83DE-8B20ECBFB843@yandex-team.ru>
<20201102195059.w6qnplbyargqwbp2@alap3.anarazel.de>
<5c832f37-db91-edec-3b31-7caef365df69@enterprisedb.com>
On 05.11.2020 22:22, Peter Eisentraut wrote:
> On 2020-11-02 20:50, Andres Freund wrote:
>> On 2020-10-31 22:25:36 +0500, Andrey Borodin wrote:
>>> But the price of compression is 1 cpu for 500MB/s (zstd). With a
>>> 20Gbps network adapters cost of recompressing all traffic is at most
>>> ~4 cores.
>>
>> It's not quite that simple, because presumably each connection is going
>> to be handled by one core at a time in the pooler. So it's easy to slow
>> down peak throughput if you also have to deal with TLS etc.
>
> Also, current deployments of connection poolers use rather small
> machine sizes. Telling users you need 4 more cores per instance now
> to decompress and recompress all the traffic doesn't seem very
> attractive. Also, it's not unheard of to have more than one layer of
> connection pooling. With that, this whole design sounds a bit like a
> heat-generation system. ;-)
Compression will be mostly useful for:
1. Replication protocol
2. COPY command
3. OLAP queries returning large result sets
4. Queries returning BLOBs/JSON
It seems to be not so good idea to switch on compression for all
connections.
And cases describe above are usually affect only small number of backends.
Also please notice, that compression may significantly reduce size of
transferred data. An copying data between multiple levels of network
protocol
also consumes significant amount of CPU. Also number of system calls can
be proportional to the size of transferred data and in many cases
now performances if servers is limited by number of system calls, rather
than by network throughput/latency and other factors.
So I accept your arguments, but still think that picture is not so
straightforward.
This is why it is very interesting to me to know results of using
compression with Odyssey in real production environment.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: k.knizhnik@postgrespro.ru, peter.eisentraut@enterprisedb.com, andres@anarazel.de, x4mmm@yandex-team.ru, pgsql-hackers@lists.postgresql.org, usernamedt@yandex-team.ru
Subject: Re: libpq compression
In-Reply-To: <a811704f-0704-f0c7-e3f8-a44aa1f082cd@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox