Subject: Re: libpq compression
To: Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: Daniil Zakhlystov <usernamedt@yandex-team.ru>,
 Andres Freund <andres@anarazel.de>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 "Andrey M. Borodin" <x4mmm@yandex-team.ru>,
 "o.bartunov@postgrespro.ru" <o.bartunov@postgrespro.ru>
References: <c0230242-2279-d6f7-e1ff-d3de908bdfbc@postgrespro.ru>
 <a5eaca1d-a226-f993-a265-8e5d8ba9bf6e@pgmasters.net>
 <c5421f2e-918a-2e74-f91a-a951d49012c4@postgrespro.ru>
 <a112e0a2-6908-5a0d-ac22-012ebb6169b0@postgrespro.ru>
 <74F0A385-6ED4-4EA7-9498-1A8527F16472@yandex-team.ru>
 <a13ce4d1-768c-1449-25f6-0cd645499d93@postgrespro.ru>
 <fa9113c0-db7a-2ed8-111c-a082a2217e4b@postgrespro.ru>
 <22446ba0-058d-624a-a679-edc0215a56c8@postgrespro.ru>
 <20201028192712.xj2qneyb3fvmjm6c@alap3.anarazel.de>
 <3b1d2752-b7b5-db39-84c9-4ff024041fff@postgrespro.ru>
 <20201030210308.54jjmnvhspk6p7cb@alap3.anarazel.de>
 <0ae2162b-818f-6a6d-cca6-ee4e73733e8e@postgrespro.ru>
 <67498D86-006D-4D56-85E2-88ABA07B2715@yandex-team.ru>
 <34328cc0-9669-f30d-50f6-46e9f3f21cde@postgrespro.ru>
 <ED4564BD-0DA3-4630-8AA3-9051D29717F7@yandex-team.ru>
 <9d33e21e-54b2-fcf3-b90f-ced286df00c5@postgrespro.ru>
 <CAEze2WiavC64KuNGKrW-M1DPb=7GiF9XshbF5CDS2ADrMJQvSQ@mail.gmail.com>
From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Message-ID: <bc017bd0-c00c-44a8-0a81-69319135446d@postgrespro.ru>
Date: Mon, 2 Nov 2020 22:20:01 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
 <CAEze2WiavC64KuNGKrW-M1DPb=7GiF9XshbF5CDS2ADrMJQvSQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Precedence: bulk


On 02.11.2020 19:53, Matthias van de Meent wrote:
> This is the result of network traffic of two backends one with enabled
>> compression and another with disable compression
>> after execution of "select * from pg_class" command:
>>
>> select * from pg_stat_network_traffic;
>>     pid  | rx_raw_bytes | tx_raw_bytes | rx_compressed_bytes |
>> tx_compressed_bytes
>> -------+--------------+--------------+---------------------+---------------------
>>    22276 |           29 |        86327 |                  38
>> |               10656
>>    22282 |           73 |        86327 |                   0
>> |                   0
> The current names and values of these columns are confusing me:
> What column contains the amount of bytes sent to/received from the
> client? Is the compression method of pid 22282 extremely efficient at
> compressing, or does it void the data (compresses down to 0 bytes)?
Names of the columns can be changed if you or somebody else will propose 
better alternatives.
This view pg_stat_network_traffic reports traffic from server (backend) 
point of view, i.e.
rx_bytes (received bytes) are commands sent from client to the server
tx_bytes (transmitted bytes) are responses sent by server to the client.

If compression is not used then rx_compressed_bytes = 
tx_compressed_bytes = 0
It seems to be more natural then assigning them the same values as (raw 
bytes).
Because it can really happen that for BLOBs with already compressed data 
(video images or sound)
compressed data will be almost the same as raw data even if compression 
is enabled.
So it seems to be important to distinguished situations when data can 
not be compressed and
when it is not compressed at all.

> I suggest having columns that contain the bytes sent to/received
> from the client before and after compression. If no compression was
> used, those numbers are expected to be equal. Example names are
> `rx_raw_bytes` and `rx_data_bytes`, `rx_received_bytes` and
> `rx_bytes_uncompressed`. Another option would be initializing /
> setting rx_compressed_bytes and tx_compressed_bytes to -1 or NULL for
> connections that do not utilize compression, to flag that
> compression is not used.
>
>
> -Matthias