Date: Mon, 25 Jun 2012 08:19:23 -0500
From: "ktm@rice.edu" <ktm@rice.edu>
To: Florian Pflug <fgp@phlo.org>
Cc: Robert Haas <robertmhaas@gmail.com>, Euler Taveira <euler@timbira.com>,
	Marko Kreen <markokr@gmail.com>, Magnus Hagander <magnus@hagander.net>,
	Martijn van Oosterhout <kleptog@svana.org>,
	Tom Lane <tgl@sss.pgh.pa.us>, Bruce Momjian <bruce@momjian.us>,
	Pgsql Hackers <pgsql-hackers@postgresql.org>,
	Claes Jakobsson <claes@versed.se>
Subject: Re: libpq compression
Message-ID: <20120625131923.GL6547@aart.rice.edu>
References: <27043.1339950593@sss.pgh.pa.us>
 <20120618174233.GA24681@svana.org>
	<CA+TgmoaTMxcAFMN8d_FXD1S_3keyq3a=Vwa7ka8cnVepy-jqjg@mail.gmail.com>
	<1D8B2B60-F42E-448C-9295-C07FACCF0611@phlo.org>
	<CABUevEybu+v7zcsM8qtOEFdQ95njyK=ERjVbdMAe3c7T2Ut6JA@mail.gmail.com>
	<7760EC03-5B38-4903-AD2D-95F787B1E1ED@phlo.org>
	<CACMqXCLkOpYnK1eLTb_bAqx344C9j=rchTeNXSW=E6+hdXwyPA@mail.gmail.com>
	<4FE49F7C.9020300@timbira.com>
	<CA+Tgmobw1Hhm4ZoS34o4bJBmMtQrnbU17v01+gCMeGmf8e06Fw@mail.gmail.com>
	<73447F47-E9A3-420B-8903-9F6A4513E229@phlo.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <73447F47-E9A3-420B-8903-9F6A4513E229@phlo.org>
User-Agent: Mutt/1.5.20 (2009-12-10)

On Mon, Jun 25, 2012 at 03:12:46PM +0200, Florian Pflug wrote:
> On Jun25, 2012, at 04:04 , Robert Haas wrote:
> > If, for
> > example, someone can demonstrate that an awesomebsdlz compresses 10x
> > as fast as OpenSSL...  that'd be pretty compelling.
> 
> That, actually, is demonstrably the case for at least Google's snappy.
> (and LZO, but that's not an option since its license is GPL) They state in
> their documentation that
> 
>   In our tests, Snappy usually is faster than algorithms in the same class
>   (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable
>   compression ratios.
> 
> The only widely supported compression method for SSL seems to be DEFLATE,
> which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib
> a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by
> an order of magnitude.
> 
> The compression ratio achieved by DEFLATE/gzip/zlib is much better, though.
> The snappy documentation states
> 
>   Typical compression ratios (based on the benchmark suite) are about
>   1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for
>   JPEGs, PNGs and other already-compressed data. Similar numbers for zlib
>   in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively.
> 
> Here are a few numbers for LZO vs. gzip. Snappy should be comparable to
> LZO - I tested LZO because I still had the command-line compressor lzop
> lying around on my machine, whereas I'd have needed to download and compile
> snappy first.
> 
> $ dd if=/dev/random of=data bs=1m count=128
> $ time gzip -1 < data > data.gz
> real	0m6.189s
> user	0m5.947s
> sys	0m0.224s
> $ time lzop < data > data.lzo
> real	0m2.697s
> user	0m0.295s
> sys	0m0.224s
> $ ls -lh data*
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:43 data
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:44 data.gz
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:44 data.lzo
> 
> $ dd if=/dev/zero of=zeros bs=1m count=128
> $ time gzip -1 < zeros > zeros.gz
> real	0m1.083s
> user	0m1.019s
> sys	0m0.052s
> $ time lzop < zeros > zeros.lzo
> real	0m0.186s
> user	0m0.123s
> sys	0m0.053s
> $ ls -lh zeros*
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:47 zeros
> -rw-r--r--  1 fgp  staff   572K Jun 25 14:47 zeros.gz
> -rw-r--r--  1 fgp  staff   598K Jun 25 14:47 zeros.lzo
> 
> To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about
> 350MB/s if the data is purely random, and about 800MB/s if the data
> compresses extremely well. (Numbers based on user time since that indicates
> the CPU time used, and ignores the IO overhead, which is substantial)
> 
> IMHO, the only compelling argument (and a very compelling one) to use
> SSL compression was that it requires very little code on our side. We've
> since discovered that it's not actually that simple, at least if we want
> to support compression without authentication or encryption, and don't
> want to restrict ourselves to using OpenSSL forever. So unless we give
> up at least one of those requirements, the arguments for using
> SSL-compression are rather thin, I think.
> 
> best regards,
> Florian Pflug
> 
+1 for http://code.google.com/p/lz4/ support. It has a BSD license too.
Using SSL libraries give all the complexity without any real benefit.

Regards,
Ken