From: ktm@rice.edu <ktm@rice.edu>
To: Florian Pflug <fgp@phlo.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Euler Taveira <euler@timbira.com>
Cc: Marko Kreen <markokr@gmail.com>
Cc: Magnus Hagander <magnus@hagander.net>
Cc: Martijn van Oosterhout <kleptog@svana.org>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: Pgsql Hackers <pgsql-hackers@postgresql.org>
Cc: Claes Jakobsson <claes@versed.se>
Subject: Re: libpq compression
Date: Mon, 25 Jun 2012 08:19:23 -0500
Message-ID: <20120625131923.GL6547@aart.rice.edu> (raw)
In-Reply-To: <73447F47-E9A3-420B-8903-9F6A4513E229@phlo.org>
References: <27043.1339950593@sss.pgh.pa.us>
<20120618174233.GA24681@svana.org>
<CA+TgmoaTMxcAFMN8d_FXD1S_3keyq3a=Vwa7ka8cnVepy-jqjg@mail.gmail.com>
<1D8B2B60-F42E-448C-9295-C07FACCF0611@phlo.org>
<CABUevEybu+v7zcsM8qtOEFdQ95njyK=ERjVbdMAe3c7T2Ut6JA@mail.gmail.com>
<7760EC03-5B38-4903-AD2D-95F787B1E1ED@phlo.org>
<CACMqXCLkOpYnK1eLTb_bAqx344C9j=rchTeNXSW=E6+hdXwyPA@mail.gmail.com>
<4FE49F7C.9020300@timbira.com>
<CA+Tgmobw1Hhm4ZoS34o4bJBmMtQrnbU17v01+gCMeGmf8e06Fw@mail.gmail.com>
<73447F47-E9A3-420B-8903-9F6A4513E229@phlo.org>
On Mon, Jun 25, 2012 at 03:12:46PM +0200, Florian Pflug wrote:
> On Jun25, 2012, at 04:04 , Robert Haas wrote:
> > If, for
> > example, someone can demonstrate that an awesomebsdlz compresses 10x
> > as fast as OpenSSL... that'd be pretty compelling.
>
> That, actually, is demonstrably the case for at least Google's snappy.
> (and LZO, but that's not an option since its license is GPL) They state in
> their documentation that
>
> In our tests, Snappy usually is faster than algorithms in the same class
> (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable
> compression ratios.
>
> The only widely supported compression method for SSL seems to be DEFLATE,
> which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib
> a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by
> an order of magnitude.
>
> The compression ratio achieved by DEFLATE/gzip/zlib is much better, though.
> The snappy documentation states
>
> Typical compression ratios (based on the benchmark suite) are about
> 1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for
> JPEGs, PNGs and other already-compressed data. Similar numbers for zlib
> in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively.
>
> Here are a few numbers for LZO vs. gzip. Snappy should be comparable to
> LZO - I tested LZO because I still had the command-line compressor lzop
> lying around on my machine, whereas I'd have needed to download and compile
> snappy first.
>
> $ dd if=/dev/random of=data bs=1m count=128
> $ time gzip -1 < data > data.gz
> real 0m6.189s
> user 0m5.947s
> sys 0m0.224s
> $ time lzop < data > data.lzo
> real 0m2.697s
> user 0m0.295s
> sys 0m0.224s
> $ ls -lh data*
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:43 data
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.gz
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.lzo
>
> $ dd if=/dev/zero of=zeros bs=1m count=128
> $ time gzip -1 < zeros > zeros.gz
> real 0m1.083s
> user 0m1.019s
> sys 0m0.052s
> $ time lzop < zeros > zeros.lzo
> real 0m0.186s
> user 0m0.123s
> sys 0m0.053s
> $ ls -lh zeros*
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:47 zeros
> -rw-r--r-- 1 fgp staff 572K Jun 25 14:47 zeros.gz
> -rw-r--r-- 1 fgp staff 598K Jun 25 14:47 zeros.lzo
>
> To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about
> 350MB/s if the data is purely random, and about 800MB/s if the data
> compresses extremely well. (Numbers based on user time since that indicates
> the CPU time used, and ignores the IO overhead, which is substantial)
>
> IMHO, the only compelling argument (and a very compelling one) to use
> SSL compression was that it requires very little code on our side. We've
> since discovered that it's not actually that simple, at least if we want
> to support compression without authentication or encryption, and don't
> want to restrict ourselves to using OpenSSL forever. So unless we give
> up at least one of those requirements, the arguments for using
> SSL-compression are rather thin, I think.
>
> best regards,
> Florian Pflug
>
+1 for http://code.google.com/p/lz4/ support. It has a BSD license too.
Using SSL libraries give all the complexity without any real benefit.
Regards,
Ken
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: ktm@rice.edu, fgp@phlo.org, robertmhaas@gmail.com, euler@timbira.com, markokr@gmail.com, magnus@hagander.net, kleptog@svana.org, tgl@sss.pgh.pa.us, bruce@momjian.us, claes@versed.se
Subject: Re: libpq compression
In-Reply-To: <20120625131923.GL6547@aart.rice.edu>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox