From: Josh Berkus <josh@agliodbs.com>
To: postgres performance list <pgsql-performance@postgresql.org>
Subject: Re: PostgreSQL as a local in-memory cache
Date: Thu, 17 Jun 2010 10:29:37 -0700
Message-ID: <4C1A5B81.8050908@agliodbs.com> (raw)
In-Reply-To: <18287446-8677-46d6-a581-c4777f60ebda@40g2000pry.googlegroups.com>
References: <cb0fb58c-9134-4314-a1d0-08fc39f911a6@40g2000pry.googlegroups.com>
<18287446-8677-46d6-a581-c4777f60ebda@40g2000pry.googlegroups.com>
All,
So, I've been discussing this because using PostgreSQL on the caching
layer has become more common that I think most people realize. Jonathan
is one of 4 companies I know of who are doing this, and with the growth
of Hadoop and other large-scale data-processing technologies, I think
demand will increase.
Especially as, in repeated tests, PostgreSQL with persistence turned off
is just as fast as the fastest nondurable NoSQL database. And it has a
LOT more features.
Now, while fsync=off and tmpfs for WAL more-or-less eliminate the IO for
durability, they don't eliminate the CPU time. Which means that a
caching version of PostgreSQL could be even faster. To do that, we'd
need to:
a) Eliminate WAL logging entirely
b) Eliminate checkpointing
c) Turn off the background writer
d) Have PostgreSQL refuse to restart after a crash and instead call an
exteral script (for reprovisioning)
Of the three above, (a) is the most difficult codewise. (b)(c) and (d)
should be relatively straightforwards, although I believe that we now
have the bgwriter doing some other essential work besides syncing
buffers. There's also a narrower use-case in eliminating (a), since a
non-fsync'd server which was recording WAL could be used as part of a
replication chain.
This isn't on hackers because I'm not ready to start working on a patch,
but I'd like some feedback on the complexities of doing (b) and (c) as
well as how many people could use a non-persistant, in-memory postgres.
--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: josh@agliodbs.com
Subject: Re: PostgreSQL as a local in-memory cache
In-Reply-To: <4C1A5B81.8050908@agliodbs.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox