Message-ID: <4C1A5B81.8050908@agliodbs.com>
Date: Thu, 17 Jun 2010 10:29:37 -0700
From: Josh Berkus <josh@agliodbs.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.9pre) Gecko/20100217 Shredder/3.0.3pre
MIME-Version: 1.0
To: postgres performance list <pgsql-performance@postgresql.org>
Subject: Re: PostgreSQL as a local in-memory cache
References: <cb0fb58c-9134-4314-a1d0-08fc39f911a6@40g2000pry.googlegroups.com>
	<18287446-8677-46d6-a581-c4777f60ebda@40g2000pry.googlegroups.com>
In-Reply-To: 
 <18287446-8677-46d6-a581-c4777f60ebda@40g2000pry.googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

All,

So, I've been discussing this because using PostgreSQL on the caching 
layer has become more common that I think most people realize.  Jonathan 
is one of 4 companies I know of who are doing this, and with the growth 
of Hadoop and other large-scale data-processing technologies, I think 
demand will increase.

Especially as, in repeated tests, PostgreSQL with persistence turned off 
is just as fast as the fastest nondurable NoSQL database.  And it has a 
LOT more features.

Now, while fsync=off and tmpfs for WAL more-or-less eliminate the IO for 
durability, they don't eliminate the CPU time.  Which means that a 
caching version of PostgreSQL could be even faster.   To do that, we'd 
need to:

a) Eliminate WAL logging entirely
b) Eliminate checkpointing
c) Turn off the background writer
d) Have PostgreSQL refuse to restart after a crash and instead call an 
exteral script (for reprovisioning)

Of the three above, (a) is the most difficult codewise.  (b)(c) and (d) 
should be relatively straightforwards, although I believe that we now 
have the bgwriter doing some other essential work besides syncing 
buffers.  There's also a narrower use-case in eliminating (a), since a 
non-fsync'd server which was recording WAL could be used as part of a 
replication chain.

This isn't on hackers because I'm not ready to start working on a patch, 
but I'd like some feedback on the complexities of doing (b) and (c) as 
well as how many people could use a non-persistant, in-memory postgres.

-- 
                                   -- Josh Berkus
                                      PostgreSQL Experts Inc.
                                      http://www.pgexperts.com