Received: from mailout06.sul.t-online.com (mailout06.sul.t-online.com [194.25.134.19]) by hub.org (8.10.1/8.10.1) with ESMTP id e66LVQW53142; Thu, 6 Jul 2000 17:31:27 -0400 (EDT) Received: from fwd01.sul.t-online.com by mailout06.sul.t-online.com with smtp id 13AJEo-0000yB-06; Thu, 06 Jul 2000 23:31:14 +0200 Received: from hot.jw.home (340000654369-0001@[62.224.105.35]) by fwd01.sul.t-online.com with esmtp id 13AJEk-1OYYTZC; Thu, 6 Jul 2000 23:31:10 +0200 Received: (from wieck@localhost) by hot.jw.home (8.8.5/8.8.5) id XAA19800; Thu, 6 Jul 2000 23:09:27 +0200 From: JanWieck@t-online.de (Jan Wieck) Message-Id: <200007062109.XAA19800@hot.jw.home> Subject: Re: [SQL] Re: lztext and compression ratios... In-Reply-To: <29094.962893375@sss.pgh.pa.us> from Tom Lane at "Jul 6, 2000 10:22:55 am" To: Tom Lane Date: Thu, 6 Jul 2000 23:09:27 +0200 (MEST) CC: PostgreSQL HACKERS , PostgreSQL GENERAL , PostgreSQL SQL Reply-To: Jan Wieck X-Mailer: ELM [version 2.4ME+ PL68 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Sender: 340000654369-0001@t-dialin.net X-Archive-Number: 200007/258 Tom Lane wrote: > JanWieck@t-online.de (Jan Wieck) writes: > >> As long as you brought it up: how sure are you that the method you've > >> used is not subject to any patents? > > > Now that you ask for it: I'm not sure. Could be. > > >> If you can show that this method uses no ideas not found in zlib, > >> then I'll feel reassured > > > To do so I don't know enough about the algorithms used in > > zlib. Is there someone out here who could verify that if I > > detailed enough describe what our compression code does? > > After a quick look at the code, I don't think there is anything > problematic about the data representation or the decompression > algorithm. The compression algorithm is another story, and it's > not real well commented :-(. The important issues are how you > search for matches in the past text and how you decide which match > is the best one to use. Please update the code comments to describe > that, and I'll take another look. Done. You'll find a new section in the top comments. While writing it I noticed that the algorithm is really expensive for big items. The history lookup table allocated is 8 times (on 32 bit architectures) the size of the input. So if you want to have 1MB compressed, it'll allocate 8MB for the history. It hit me when I was hunting a bug in the toaster earlier today. Doing an update to a toasted item of 5MB, resulting in a new value of 10MB, the backend blew up to 290MB of virtual memory - oh boy. I definitely need to make that smarter. When I wrote it I never thought about items that big. It was before we had the idea of TOAST. This all might open another discussion I'll start in a separate thread. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #