Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fWOZQ-00020e-14 for pgsql-hackers@arkaria.postgresql.org; Fri, 22 Jun 2018 16:05:56 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fWOZN-0002fG-Sp for pgsql-hackers@arkaria.postgresql.org; Fri, 22 Jun 2018 16:05:53 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fWOZN-0002f9-4d for pgsql-hackers@lists.postgresql.org; Fri, 22 Jun 2018 16:05:53 +0000 Received: from homie-sub4.mail.dreamhost.com ([69.163.253.135] helo=homiemail-a132.g.dreamhost.com) by makus.postgresql.org with esmtps (TLS1.1:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fWOZJ-0001K5-12 for pgsql-hackers@lists.postgresql.org; Fri, 22 Jun 2018 16:05:50 +0000 Received: from homiemail-a132.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a132.g.dreamhost.com (Postfix) with ESMTP id ED8C130002BA2; Fri, 22 Jun 2018 09:05:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cryptonector.com; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=cryptonector.com; bh=HTgfBS93Vw2vcs HZFGMZUarNenE=; b=OtY9wCEc0HwZlU3Ex9YRxp/kKvGCuEAhAjoBUWyHnABqRe 3cUeiC9GM+o9VEeK9ODrtsemfpm400LkfoFn/UvSPRHNH8rjsuopygeQhs3rBVjz HLjejo2319Gvt/EnpasEHOnJn9We2K6y6tq8LHZOwi2sWJ94MbactlrsWsv5M= Received: from localhost (cpe-70-123-158-140.austin.res.rr.com [70.123.158.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: nico@cryptonector.com) by homiemail-a132.g.dreamhost.com (Postfix) with ESMTPSA id 1D90630002B96; Fri, 22 Jun 2018 09:05:45 -0700 (PDT) Date: Fri, 22 Jun 2018 11:05:43 -0500 From: Nico Williams To: Konstantin Knizhnik Cc: Robbie Harwood , Thomas Munro , Grigory Smolkin , PostgreSQL Hackers Subject: Re: libpq compression Message-ID: <20180622160543.GM4200@localhost> References: <3b0260da-fe47-2341-da01-51383fe9172d@postgrespro.ru> <15419613-35b5-0ff6-6261-834abf73330c@postgrespro.ru> <9b3a735c-06c2-631f-8e97-e3982e575210@postgrespro.ru> <20180621213404.GK4200@localhost> <6eee9b98-0d77-c6a4-c800-6a9332c6010d@postgrespro.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6eee9b98-0d77-c6a4-c800-6a9332c6010d@postgrespro.ru> User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote: > On 22.06.2018 00:34, Nico Williams wrote: > >So I think you just have to have lengths. > > > >Now, this being about compression, I understand that you might now want > >to have 4-byte lengths, especially given that most messages will be > >under 8KB. So use a varint encoding for the lengths. > > No explicit framing and lengths are needed in case of using streaming > compression. > There can be certainly some kind of frames inside compression protocol > itself, but it is intrinsic of compression algorithm. I don't think that's generally true. It may be true of the compression algorithm you're working with. This is fine, of course, but plugging in other compression algorithms will require the authors to add framing.