pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Nico Williams <nico@cryptonector.com>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Robbie Harwood <rharwood@redhat.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: Grigory Smolkin <g.smolkin@postgrespro.ru>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: libpq compression
Date: Thu, 21 Jun 2018 16:34:06 -0500
Message-ID: <20180621213404.GK4200@localhost> (raw)
In-Reply-To: <9b3a735c-06c2-631f-8e97-e3982e575210@postgrespro.ru>
References: <CAEepm=3CGUX4Vyw=jFaOedt99HS1HQiBKYTX4OYHo7HU2AczZg@mail.gmail.com>
	<36db5d02-45e6-b863-1418-825ef20ae4c6@postgrespro.ru>
	<CAEepm=3RAS8xVQK8rs+Xitzp+JXzGdGsMA7f+H8fjKQ7Do8aSA@mail.gmail.com>
	<f4ad0809-dd90-e321-5c1a-0ca68a136a8f@postgrespro.ru>
	<jlg4lhzdeak.fsf@redhat.com>
	<3b0260da-fe47-2341-da01-51383fe9172d@postgrespro.ru>
	<jlgzhzqbi84.fsf@redhat.com>
	<15419613-35b5-0ff6-6261-834abf73330c@postgrespro.ru>
	<jlgo9g5qjst.fsf@redhat.com>
	<9b3a735c-06c2-631f-8e97-e3982e575210@postgrespro.ru>

On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote:
> On 20.06.2018 23:34, Robbie Harwood wrote:
> >Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
> >Well, that's a design decision you've made.  You could put lengths on
> >chunks that are sent out - then you'd know exactly how much is needed.
> >(For instance, 4 bytes of network-order length followed by a complete
> >payload.)  Then you'd absolutely know whether you have enough to
> >decompress or not.
> 
> Do you really suggest to send extra header for each chunk of data?
> Please notice that chunk can be as small as one message: dozen of bytes
> because libpq is used for client-server communication with request-reply
> pattern.

You must have lengths, yes, otherwise you're saying that the chosen
compression mechanism must itself provide framing.

I'm not that familiar with compression APIs and formats, but looking at
RFC1950 (zlib) for example I see no framing.

So I think you just have to have lengths.

Now, this being about compression, I understand that you might now want
to have 4-byte lengths, especially given that most messages will be
under 8KB.  So use a varint encoding for the lengths.

Nico
-- 




view thread (289+ messages)  latest in thread

Message-ID: <20180621213404.GK4200@localhost>
Permalink:  ../20180621213404.GK4200@localhost/
Also on:    postgresql.org/message-id/20180621213404.GK4200@localhost

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: nico@cryptonector.com, k.knizhnik@postgrespro.ru, rharwood@redhat.com, thomas.munro@enterprisedb.com, g.smolkin@postgrespro.ru, pgsql-hackers@lists.postgresql.org
  Subject: Re: libpq compression
  In-Reply-To: <20180621213404.GK4200@localhost>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox