Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fW7Dd-0000TX-RZ for pgsql-hackers@arkaria.postgresql.org; Thu, 21 Jun 2018 21:34:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fW7Db-0005yv-LH for pgsql-hackers@arkaria.postgresql.org; Thu, 21 Jun 2018 21:34:15 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fW7Da-0005yo-SF for pgsql-hackers@lists.postgresql.org; Thu, 21 Jun 2018 21:34:15 +0000 Received: from homie-sub4.mail.dreamhost.com ([69.163.253.135] helo=homiemail-a132.g.dreamhost.com) by makus.postgresql.org with esmtps (TLS1.1:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fW7DX-0002Ge-0c for pgsql-hackers@lists.postgresql.org; Thu, 21 Jun 2018 21:34:13 +0000 Received: from homiemail-a132.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a132.g.dreamhost.com (Postfix) with ESMTP id DF8CC30002904; Thu, 21 Jun 2018 14:34:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cryptonector.com; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=cryptonector.com; bh=bDrnYzyB+xkTZr FR8FdgNaObEjA=; b=lQpvMsScL8uC+XsmRW1LpQHxL2fko6d2cdjnnVtCf1AvBY KxwhuNWRWqykkuAUBhRHHWxF1r2CuZ2XJWCoHAiOkiKz04KWbEOXE23eYxWNtLAw NehlrL/DN2YbCyWMcWnipclIrqryCcnllbqcqMxkecgOncOYM4p20xCiemAdY= Received: from localhost (cpe-70-123-158-140.austin.res.rr.com [70.123.158.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: nico@cryptonector.com) by homiemail-a132.g.dreamhost.com (Postfix) with ESMTPSA id 66F6030002900; Thu, 21 Jun 2018 14:34:08 -0700 (PDT) Date: Thu, 21 Jun 2018 16:34:06 -0500 From: Nico Williams To: Konstantin Knizhnik Cc: Robbie Harwood , Thomas Munro , Grigory Smolkin , PostgreSQL Hackers Subject: Re: libpq compression Message-ID: <20180621213404.GK4200@localhost> References: <36db5d02-45e6-b863-1418-825ef20ae4c6@postgrespro.ru> <3b0260da-fe47-2341-da01-51383fe9172d@postgrespro.ru> <15419613-35b5-0ff6-6261-834abf73330c@postgrespro.ru> <9b3a735c-06c2-631f-8e97-e3982e575210@postgrespro.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9b3a735c-06c2-631f-8e97-e3982e575210@postgrespro.ru> User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote: > On 20.06.2018 23:34, Robbie Harwood wrote: > >Konstantin Knizhnik writes: > >Well, that's a design decision you've made. You could put lengths on > >chunks that are sent out - then you'd know exactly how much is needed. > >(For instance, 4 bytes of network-order length followed by a complete > >payload.) Then you'd absolutely know whether you have enough to > >decompress or not. > > Do you really suggest to send extra header for each chunk of data? > Please notice that chunk can be as small as one message: dozen of bytes > because libpq is used for client-server communication with request-reply > pattern. You must have lengths, yes, otherwise you're saying that the chosen compression mechanism must itself provide framing. I'm not that familiar with compression APIs and formats, but looking at RFC1950 (zlib) for example I see no framing. So I think you just have to have lengths. Now, this being about compression, I understand that you might now want to have 4-byte lengths, especially given that most messages will be under 8KB. So use a varint encoding for the lengths. Nico --