From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
To: Craig Ringer <craig.ringer@enterprisedb.com>
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>
Cc: Konstantin Knizhnik <knizhnik@garret.ru>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: pryzby@telsasoft.com, x4mmm@yandex-team.ru, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: libpq compression
Date: Tue, 23 Feb 2021 19:29:12 +0300
Message-ID: <a364948a-d93b-fe17-f62d-334f34360daa@postgrespro.ru> (raw)
In-Reply-To: <CAGRY4nxpvFOHeq=xOe0L8chu3eG1aMK71kRwWM-q7CQUEcqwCQ@mail.gmail.com>
References: <aad16e41-b3f9-e89d-fa57-fb4c694bec25@postgrespro.ru>
<612f227d-cb27-9238-5922-1500f8b2eb96@postgrespro.ru>
<160623927937.7563.15974279758478581796.pgcf@coridan.postgresql.org>
<CA+TgmoY5AMq8auemc0g3geCpVDeRnawW1A-jUeqT-5cE-_2MiA@mail.gmail.com>
<6811D196-E2FB-40CC-B1C9-F19427FBF675@yandex-team.ru>
<CA+TgmoZMTLydjP3iWBir20OrSXTugxS2CXC1WP1nWM5=Zqz0Nw@mail.gmail.com>
<A1D0D9A4-1FC4-4311-83C7-EEA90749EE07@yandex-team.ru>
<CA+TgmoZrM9T_W9YR-vLYZXj-vC_+F-DER65A87+pDXRwUFjJXg@mail.gmail.com>
<D5354E7A-3B9F-4D32-B3AB-F65058D36500@yandex-team.ru>
<CA+TgmobPjqe-89PRH_qGHMWruXPK-Ya9tABEFuCvv1j26m5Efw@mail.gmail.com>
<470E411E-681D-46A2-A1E9-6DE11B5F59F3@yandex-team.ru>
<3676BA4F-ED2B-470C-9892-007C943BB1D2@yandex-team.ru>
<CAGRY4nxpvFOHeq=xOe0L8chu3eG1aMK71kRwWM-q7CQUEcqwCQ@mail.gmail.com>
On 22.02.2021 08:38, Craig Ringer wrote:
>
>
> On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov,
> <usernamedt@yandex-team.ru <mailto:usernamedt@yandex-team.ru>> wrote::
>
>
> 3. Chunked compression allows to compress only well compressible
> messages and save the CPU cycles by not compressing the others
> 4. Chunked compression introduces some traffic overhead compared
> to the permanent (1.2810G vs 1.2761G TX data on pg_restore of IMDB
> database dump, according to results in my previous message)
> 5. From the protocol point of view, chunked compression seems a
> little bit more flexible:
> - we can inject some uncompressed messages at any time without
> the need to decompress/compress the compressed data
> - we can potentially switch the compression algorithm at any time
> (but I think that this might be over-engineering)
>
>
> Chunked compression also potentially makes it easier to handle non
> blocking sockets, because you aren't worrying about yet another layer
> of buffering within the compression stream. This is a real pain with
> SSL, for example.
>
> It simplifies protocol analysis.
>
> It permits compression to be decided on the fly, heuristically, based
> on message size and potential compressibility.
>
> It could relatively easily be extended to compress a group of pending
> small messages, e.g. by PQflush. That'd help mitigate the downsides
> with small messages.
>
> So while stream compression offers better compression ratios, I'm
> inclined to suspect we'll want message level compression.
From my point of view there are several use cases where protocol
compression can be useful for:
1. Replication
2. Backup/dump
3. Bulk load (COPY)
4. Queries returning large objects (json, blobs,...)
All this cases are controlled by user or DBA, so them can make decision
whether to use compression or not.
Switching compression on the fly, use different algorithms in different
directions is not needed.
Yes, in all this scenarios data is mostly transferred in one direction.
So compression of small messages going in opposite direction is not
strictly needed.
But benchmarks shows that is has almost no influence on performance and
CPU usage.
So I suggest not to complicate protocol and implementation and implement
functionality which is present in most of other DBMSes.
There is no sense to try compression on workloads like pgbench and make
some conclusions based on it. From my point of view it is obvious misuse.
Compressing of each message idividually or chunked compression
significantly decrease compression ratio because typical size of message
is not large enough
and resetting compression state after processing each message (clearing
compression dictionary) adds too large overhead.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: k.knizhnik@postgrespro.ru, craig.ringer@enterprisedb.com, usernamedt@yandex-team.ru, knizhnik@garret.ru, robertmhaas@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: libpq compression
In-Reply-To: <a364948a-d93b-fe17-f62d-334f34360daa@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox