X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 5027ED1B269 for ; Sat, 10 Jul 2004 10:59:03 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 17952-03 for ; Sat, 10 Jul 2004 13:59:03 +0000 (GMT) Received: from news.hub.org (news.hub.org [200.46.204.72]) by svr1.postgresql.org (Postfix) with ESMTP id 6E931D1B267 for ; Sat, 10 Jul 2004 10:58:53 -0300 (ADT) Received: from news.hub.org (news.hub.org [200.46.204.72]) by news.hub.org (8.12.9/8.12.9) with ESMTP id i6ADwwi7064572 for ; Sat, 10 Jul 2004 13:58:58 GMT (envelope-from news@news.hub.org) Received: (from news@localhost) by news.hub.org (8.12.9/8.12.9/Submit) id i6ADoWlk063629 for pgsql-performance@postgresql.org; Sat, 10 Jul 2004 13:50:32 GMT From: Christopher Browne X-Newsgroups: comp.databases.postgresql.performance Subject: Re: Working on huge RAM based datasets Date: Sat, 10 Jul 2004 09:06:34 -0400 Organization: cbbrowne Computing Inc Lines: 44 Message-ID: References: <20040708101913.GA15871@uio.no> <00ac01c46516$3ec859e0$0300a8c0@lappy> <40EDF7DE.8080900@familyhealth.com.au> <011301c46597$15d145c0$0300a8c0@lappy> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Complaints-To: usenet@news.hub.org X-message-flag: Outlook is rather hackable, isn't it? X-Home-Page: http://www.cbbrowne.com/info/ X-Affero: http://svcs.affero.net/rm.php?r=cbbrowne User-Agent: Gnus/5.1006 (Gnus v5.10.6) XEmacs/21.4 (Security Through Obscurity, linux) Cancel-Lock: sha1:J0dE4S02h10Xyja2/mYapvMvZpU= To: pgsql-performance@postgresql.org X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200407/58 X-Sequence-Number: 7449 Quoth andy_ballingall@bigfoot.com ("Andy Ballingall"): > This is the future, isn't it? Each year, a higher percentage of DB > applications will be able to fit entirely in RAM, and that percentage is > going to be quite significant in just a few years. The disk system gets > relegated to a data preload on startup and servicing the writes as the > server does its stuff. Regrettably, this may be something that fits better with MySQL, as it already has an architecture oriented to having different "storage engines" in behind. There may be merit to the notion of implementing in-memory databases; some assumptions change: - You might use bitmap indices, although that probably "kills" MVCC; - You might use T-trees rather than B-trees for indices, although research seems to indicate that B-trees win out if there is a great deal of concurrent access; - It can become worthwhile to use compression schemes to fit more records into memory that wouldn't be worthwhile if using demand paging. If you really want to try this, then look at Konstantin Knizhnik's FastDB system: http://www.ispras.ru/~knizhnik/fastdb.html It assumes that your application will be a monolithic C++ process; if that isn't the case, then performance will probably suffer due to throwing in context switches. The changes in assumptions are pretty vital ones, that imply you're heading in a fairly different direction than that which PostgreSQL seems to be taking. That's not to say that there isn't merit to building a database system using T-trees and bitmap indices attuned to applications where main-memory storage is key; it's just that the proposal probably should go somewhere else. -- output = ("cbbrowne" "@" "cbbrowne.com") http://www3.sympatico.ca/cbbrowne/languages.html How does the guy who drives the snowplow get to work in the mornings?