Message-ID: <4C1881C7.1050501@catalyst.net.nz>
Date: Wed, 16 Jun 2010 19:48:23 +1200
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4
MIME-Version: 1.0
To: pgsql-performance@postgresql.org
Subject: Re: PostgreSQL as a local in-memory cache
References: <cb0fb58c-9134-4314-a1d0-08fc39f911a6@40g2000pry.googlegroups.com>
	<87bpbcpabg.fsf@cbbrowne-laptop.afilias-int.info>
	<4C17A417.2040600@noao.edu>
	<87ocfcnqmk.fsf@cbbrowne-laptop.afilias-int.info>
	<4C180A3B.7090905@agliodbs.com>
	<4634b12c-749f-4605-9d95-121b8e2bd9e4@k17g2000pro.googlegroups.com>
In-Reply-To: 
 <4634b12c-749f-4605-9d95-121b8e2bd9e4@k17g2000pro.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 16/06/10 18:30, jgardner@jonathangardner.net wrote:
> On Jun 15, 4:18 pm, j...@agliodbs.com (Josh Berkus) wrote:
>    
>> On 6/15/10 10:37 AM, Chris Browne wrote:
>>
>> I'd like to see some figures about WAL on RAMfs vs. simply turning off
>> fsync and full_page_writes.  Per Gavin's tests, PostgreSQL is already
>> close to TokyoCabinet/MongoDB performance just with those turned off; I
>> wonder if actually having the WAL on a memory partition would make any
>> real difference in throughput.
>>
>> I've seen a lot of call for this recently, especially since PostgreSQL
>> seems to be increasingly in use as a reporting server for Hadoop.  Might
>> be worth experimenting with just making wal writing a no-op.  We'd also
>> want to disable checkpointing, of course.
>>
>>      
> My back-of-the-envelope experiment: Inserting single integers into a
> table without indexes using a prepared query via psycopg2.
>
> Python Script:
> import psycopg2
> from time import time
> conn = psycopg2.connect(database='jgardner')
> cursor = conn.cursor()
> cursor.execute("CREATE TABLE test (data int not null)")
> conn.commit()
> cursor.execute("PREPARE ins AS INSERT INTO test VALUES ($1)")
> conn.commit()
> start = time()
> tx = 0
> while time() - start<  1.0:
>      cursor.execute("EXECUTE ins(%s)", (tx,));
>      conn.commit()
>      tx += 1
> print tx
> cursor.execute("DROP TABLE test");
> conn.commit();
>
> Local disk, WAL on same FS:
> * Default config =>  90
> * full_page_writes=off =>  90
> * synchronous_commit=off =>  4,500
> * fsync=off =>  5,100
> * fsync=off and synchronous_commit=off =>  5,500
> * fsync=off and full_page_writes=off =>  5,150
> * fsync=off, synchronous_commit=off and full_page_writes=off =>  5,500
>
> tmpfs, WAL on same tmpfs:
> * Default config: 5,200
> * full_page_writes=off =>  5,200
> * fsync=off =>  5,250
> * synchronous_commit=off =>  5,200
> * fsync=off and synchronous_commit=off =>  5,450
> * fsync=off and full_page_writes=off =>  5,250
> * fsync=off, synchronous_commit=off and full_page_writes=off =>  5,500
>
> NOTE: If I do one giant commit instead of lots of littler ones, I get
> much better speeds for the slower cases, but I never exceed 5,500
> which appears to be some kind of wall I can't break through.
>
> If there's anything else I should tinker with, I'm all ears.
>
>    

Seeing some profiler output (e.g oprofile) for the fastest case (and 
maybe 'em all later) might be informative about what limit is being hit 
here.

regards

Mark