From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
To: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: libpq compression
Date: Tue, 15 May 2018 15:53:36 +0300
Message-ID: <93107541-1bec-d5f2-82ec-05b7afa4488a@postgrespro.ru> (raw)
In-Reply-To: <CA+q6zcX4SZeJgO-55q+YmBDJM6mrU3ECPFBX=yEme9oNoN+sZw@mail.gmail.com>
References: <aad16e41-b3f9-e89d-fa57-fb4c694bec25@postgrespro.ru>
<CA+q6zcX4SZeJgO-55q+YmBDJM6mrU3ECPFBX=yEme9oNoN+sZw@mail.gmail.com>
On 15.05.2018 13:23, Dmitry Dolgov wrote:
> > On 30 March 2018 at 14:53, Konstantin Knizhnik
> <k.knizhnik@postgrespro.ru <mailto:k.knizhnik@postgrespro.ru>> wrote:
> > Hi hackers,
> > One of our customers was managed to improve speed about 10 times by
> using SSL compression for the system where client and servers are
> located in different geographical regions
> > and query results are very large because of JSON columns. Them
> actually do not need encryption, just compression.
> > I expect that it is not the only case where compression of libpq
> protocol can be useful. Please notice that Postgres replication is
> also using libpq protocol.
> >
> > Taken in account that vulnerability was found in SSL compression and
> so SSLComppression is considered to be deprecated and insecure
> (http://www.postgresql-archive.org/disable-SSL-compression-td6010072.html),
> it will be nice to have some alternative mechanism of reducing libpq
> traffic.
> >
> > I have implemented some prototype implementation of it (patch is
> attached).
> > To use zstd compression, Postgres should be configured with
> --with-zstd. Otherwise compression will use zlib unless it is disabled
> by --without-zlib option.
> > I have added compression=on/off parameter to connection string and
> -Z option to psql and pgbench utilities.
>
> I'm a bit confused why there was no reply to this. I mean, it wasn't
> sent on
> 1st April, the patch still can be applied on top of the master branch
> and looks
> like it even works.
>
> I assume the main concern her is that it's implemented in a rather not
> extensible way. Also, if I understand correctly, it compresses the
> data stream
> in both direction server <-> client, not sure if there is any value in
> compressing what a client sends to a server. But still I'm wondering
> why it
> didn't start at least a discussion about how it can be implemented. Do
> I miss
> something?
Implementation of libpq compression will be included in next release of
PgProEE.
Looks like community is not so interested in this patch. Frankly
speaking I do not understand why.
Compression of libpq traffic can significantly increase speed of:
1. COPY
2. Replication (both streaming and logical)
3. Queries returning large results sets (for example JSON) through slow
connections.
It is possible to compress libpq traffic using SSL compression. But SSL
compression is unsafe and deteriorated feature.
Yes, this patch is not extensible: it can use either zlib either zstd.
Unfortunately internal Postgres compression pglz doesn't provide
streaming API.
May be it is good idea to combine it with Ildus patch (custom
compression methods): https://commitfest.postgresql.org/18/1294/
In this case it will be possible to use any custom compression
algorithm. But we need to design and implement streaming API for pglz
and other compressors.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: k.knizhnik@postgrespro.ru, 9erthalion6@gmail.com
Subject: Re: libpq compression
In-Reply-To: <93107541-1bec-d5f2-82ec-05b7afa4488a@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox