From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
To: pgsql-performance@postgresql.org
Subject: Re: PostgreSQL as a local in-memory cache
Date: Wed, 16 Jun 2010 19:48:23 +1200
Message-ID: <4C1881C7.1050501@catalyst.net.nz> (raw)
In-Reply-To: <4634b12c-749f-4605-9d95-121b8e2bd9e4@k17g2000pro.googlegroups.com>
References: <cb0fb58c-9134-4314-a1d0-08fc39f911a6@40g2000pry.googlegroups.com>
<87bpbcpabg.fsf@cbbrowne-laptop.afilias-int.info>
<4C17A417.2040600@noao.edu>
<87ocfcnqmk.fsf@cbbrowne-laptop.afilias-int.info>
<4C180A3B.7090905@agliodbs.com>
<4634b12c-749f-4605-9d95-121b8e2bd9e4@k17g2000pro.googlegroups.com>
On 16/06/10 18:30, jgardner@jonathangardner.net wrote:
> On Jun 15, 4:18 pm, j...@agliodbs.com (Josh Berkus) wrote:
>
>> On 6/15/10 10:37 AM, Chris Browne wrote:
>>
>> I'd like to see some figures about WAL on RAMfs vs. simply turning off
>> fsync and full_page_writes. Per Gavin's tests, PostgreSQL is already
>> close to TokyoCabinet/MongoDB performance just with those turned off; I
>> wonder if actually having the WAL on a memory partition would make any
>> real difference in throughput.
>>
>> I've seen a lot of call for this recently, especially since PostgreSQL
>> seems to be increasingly in use as a reporting server for Hadoop. Might
>> be worth experimenting with just making wal writing a no-op. We'd also
>> want to disable checkpointing, of course.
>>
>>
> My back-of-the-envelope experiment: Inserting single integers into a
> table without indexes using a prepared query via psycopg2.
>
> Python Script:
> import psycopg2
> from time import time
> conn = psycopg2.connect(database='jgardner')
> cursor = conn.cursor()
> cursor.execute("CREATE TABLE test (data int not null)")
> conn.commit()
> cursor.execute("PREPARE ins AS INSERT INTO test VALUES ($1)")
> conn.commit()
> start = time()
> tx = 0
> while time() - start< 1.0:
> cursor.execute("EXECUTE ins(%s)", (tx,));
> conn.commit()
> tx += 1
> print tx
> cursor.execute("DROP TABLE test");
> conn.commit();
>
> Local disk, WAL on same FS:
> * Default config => 90
> * full_page_writes=off => 90
> * synchronous_commit=off => 4,500
> * fsync=off => 5,100
> * fsync=off and synchronous_commit=off => 5,500
> * fsync=off and full_page_writes=off => 5,150
> * fsync=off, synchronous_commit=off and full_page_writes=off => 5,500
>
> tmpfs, WAL on same tmpfs:
> * Default config: 5,200
> * full_page_writes=off => 5,200
> * fsync=off => 5,250
> * synchronous_commit=off => 5,200
> * fsync=off and synchronous_commit=off => 5,450
> * fsync=off and full_page_writes=off => 5,250
> * fsync=off, synchronous_commit=off and full_page_writes=off => 5,500
>
> NOTE: If I do one giant commit instead of lots of littler ones, I get
> much better speeds for the slower cases, but I never exceed 5,500
> which appears to be some kind of wall I can't break through.
>
> If there's anything else I should tinker with, I'm all ears.
>
>
Seeing some profiler output (e.g oprofile) for the fastest case (and
maybe 'em all later) might be informative about what limit is being hit
here.
regards
Mark
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: mark.kirkwood@catalyst.net.nz
Subject: Re: PostgreSQL as a local in-memory cache
In-Reply-To: <4C1881C7.1050501@catalyst.net.nz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox