Message-ID: <4CF74F99.4050008@enterprisedb.com>
Date: Thu, 02 Dec 2010 09:49:45 +0200
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI;
	rv:1.9.1.15) Gecko/20101030 Icedove/3.0.10
MIME-Version: 1.0
To: Joachim Wieland <joe@mcknight.de>
CC: Tom Lane <tgl@sss.pgh.pa.us>, =?ISO-8859-1?Q?Jos=E9_Arthur_Beneta?=
	=?ISO-8859-1?Q?sso_Villanova?= <jose.arthur@gmail.com>,
	pgsql-hackers@postgresql.org
Subject: Re: directory archive format for pg_dump
References: <AANLkTi=AEi3F6jC0Zy3dG9njiwJJC8Y6V090-hMonyyn@mail.gmail.com>
	<AANLkTik+PR2HCsqS7PVAdETPyKSGn2hb-eTeg9QZpRew@mail.gmail.com>
	<4CEA9F29.5060106@enterprisedb.com>	<28890.1290445623@sss.pgh.pa.us>
	<4CEAD61A.6060207@enterprisedb.com>
	<AANLkTi=neaBqvLa2ZnDpuAMSen3P4ZHE8__Q7L2pmjD0@mail.gmail.com>
	<4CF40B5F.1010407@enterprisedb.com>	<4CF655A3.4090308@enterprisedb.com>
	<4CF6562E.60506@enterprisedb.com>
	<AANLkTinauQsCGt-yeyeN_QWuN+tao+q126rwKnO2QWLP@mail.gmail.com>
In-Reply-To: <AANLkTinauQsCGt-yeyeN_QWuN+tao+q126rwKnO2QWLP@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 02.12.2010 04:35, Joachim Wieland wrote:
> There is one thing however that I am not in favor of, which is the
> removal of the "sizeHint" parameter for the read functions. The reason
> for this parameter is not very clear now without LZF but I have tried
> to put in a few comments to explain the situation (which you have
> taken out as well :-) ).
>
> The point is that zlib is a stream based compression algorithm, you
> just stuff data in and from time to time you get data out and in the
> end you explicitly flush the compressor. The read function can just
> return as many bytes as it wants and we can just hand it all over to
> zlib. Other compression algorithms however are block based and first
> write a block header that contains the information on the next data
> block, including uncompressed and compressed sizes. Now with the
> sizeHint parameter I used, the compressor could tell the read function
> that it just wants to read the fixed size header (6 bytes IIRC). In
> the header it would look up the compressed size for the next block and
> would then ask the read function to get exactly this amount of data,
> decompress it and go on with the next block, and so forth...
>
> Of course you can possibly do that memory management inside the
> compressor with an extra buffer holding what you got in excess but
> it's a pain. If you removed that part on purpose on the grounds that
> there is no block based compression algorithm in core and probably
> never will be, then that's okay :-)

Yeah, we're not going to have lzf built-in anytime soon. The external 
command approach seems like the best way to support additional 
compression algorithms, and I don't think it could do anything with 
sizeHint. And the custom format didn't obey sizeHint anyway, because it 
reads one custom-format block at a time.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com